This content originally appeared on DEV Community and was authored by XINZ
As a renowned technical expert in the DBA (Database Administrator) community, Xu Ji has remained active in the industry, having been a speaker at numerous database and IT operations events, and has gained a large following. Xu Ji’s works continue to be essential reading for many DBAs, with numerous professionals finding solutions to their problems in his books. Despite his fame in the database field, Xu Ji originally entered the field somewhat by chance. How did he gradually become a database expert?
One day in 2015, Xu Ji received an urgent call for help from State Grid Corporation of China (SGCC). It turned out that the State Grid was under attack by an Oracle database SCN HEADROOM storm, and the SCN ceiling was less than 10 days away from being reached. If the source of the issue wasn’t identified in time, many of SGCC’s Oracle databases could crash when the storm hit. This would affect core systems in over 20 provinces and cities under SGCC, causing immeasurable losses.
Urgent Call for Help
SCN (System Change Number) is an internal logical clock used by Oracle to distinguish the order of transaction operations and ensure the consistency of these operations. It is made up of a 48-bit one-way sequence number. To ensure this 48-bit SCN could be used for a long time, Oracle had imposed a restriction: the SCN should not increase more than 16K per second. Under normal circumstances, this increment would be sufficient, but Oracle had a bug. In certain situations, this bug would cause the SCN to increase rapidly, quickly approaching the ceiling. Once the limit was reached, it could cause the database to crash for protection.
The Oracle bug had been around for years, and Oracle had released patches. However, if even one database didn’t have the patch, SCN synchronization would occur when other databases connected via DBLink. This was the situation SGCC faced at the time. Despite most of SGCC’s Oracle databases being patched, SCN HEADROOM continued to decrease abnormally, suggesting that a few databases had not been patched. The urgent task was to locate the unpatched database before the SCN reached the ceiling, take it offline, or patch it immediately to resolve the issue.
This was the tough situation Xu Ji found himself in. SGCC set up a video command center for Xu Ji and authorized him to command and coordinate the data units under SGCC. The pressure was immense — time was tight, and this was an extremely important issue, with no room for error.
“SGCC’s business spans 27 provinces, with over 30 data units and thousands of Oracle databases. In such a complex network environment, it’s hard to tell who the source is and who is infected. It had to be located in just a few days, or it could lead to major operational accidents,” Xu Ji said.
Fortunately, Xu Ji had dealt with similar situations with telecom operators in the past. Upon receiving the task, Xu Ji quickly arranged for scripts to be written, then distributed them across SGCC’s systems. A few hours later, the data came in. Xu Ji had dozens of people working overnight, and about 24 hours later, they identified the problem to be in a system in the northwest, another unit’s database deployed within SGCC. This database, not owned by SGCC, hadn’t been patched, triggering the SCN crisis.
After more than two days, Xu Ji hadn’t left the command center, resting only briefly in the hall when he got tired. It’s easy to imagine the immense sense of pride Xu Ji must have felt when the problem was accurately pinpointed. As a technical professional, having the opportunity and capability to contribute back to society and the country was the best reward for his efforts, a moment he would remember for a lifetime.
Turning to Databases: The Breakthrough Moment
Although Xu Ji is now a well-established database expert, he didn’t start out specializing in databases. It was a field he turned to later in his career. In fact, before 2000 and in the early years after, Xu Ji was more famous in the software field than in databases.
Xu Ji graduated from Nanjing University in 1992 with a degree in computer science. Back then, computers were still rare; only large companies had them, and they were kept in dedicated computer rooms. Computer professionals were highly valuable. Xu Ji was not the type to settle down, having worked for several companies before venturing into entrepreneurship. While his work often involved databases and database optimization and maintenance, it was more of a side role, with his main focus being software development and system optimization, where he was quite well-known in the industry.
In 2006, the company Xu Ji worked for became an outsourcing partner for Oracle in China. This led to more frequent interactions with Oracle databases, and Xu Ji’s focus gradually shifted toward databases, quickly establishing himself as an expert and gaining significant recognition within the industry. By 2011, SGCC faced performance issues with several projects deployed a few years earlier due to lack of proper maintenance, and they were looking for optimization experts across the country. Xu Ji, with his reputation, was selected by SGCC for the task.
“Back then, a leader in charge of the project at SGCC was supposed to meet with me. He was very busy, so I flew from Shenzhen to meet him. I waited for him the whole morning, but he couldn’t find the time, so he decided to talk while having lunch. During the lunch, I presented my plan, and the leader was very interested. He canceled his subsequent schedule, and we continued our discussion until 2 PM,” Xu Ji recalled.
This incident not only highlighted SGCC’s emphasis on the project but also demonstrated that Xu Ji’s solution was feasible. Shortly after returning to Shenzhen, Xu Ji received a notice from SGCC that the project would move forward. The result was outstanding, with the system’s performance improving by over 10 times. The project involved much more than just database optimization — it encompassed the overall optimization of both hardware and software environments, including core algorithm improvements in application software.
This project made Xu Ji famous within SGCC, and he later participated in many more projects with them. As his reputation grew, more similar projects reached out to him. Xu Ji remembers a particular project where the system, only two years after going live, had become increasingly slow. The company was about to invest 35 million yuan to expand the system. Hearing about Xu Ji’s expertise, they asked him if he could help optimize the system, or at least reduce the expansion cost. Xu Ji and his team worked for more than a month and, in the end, assured the leaders that the system could last for another five years without the need for expansion.
“Actually, the work we did for the 35 million yuan project wasn’t very complicated — mainly organizing fragments, setting virtual volumes, adjusting table partitions, and other routine tasks. Originally, we had planned to optimize the SQL queries too, but the improvements from the earlier tasks were so significant that we didn’t need to go further,” Xu Ji explained.
With each successful project, Xu Ji became a well-known figure within SGCC. One leader even told his subordinates, “You’d better keep an eye on Xu Ji, don’t let him slip away — this person is valuable.”
Growing with Domestic Databases
After 2000, Xu Ji gradually shifted his focus entirely to databases, particularly after his company became Oracle’s outsourcing partner in 2006. As he aged and his ability to write code slowed down, Xu Ji also hoped to focus more on database maintenance. “After all, working with databases is a profession you can continue in until old age,” he said.
Having long been engaged in database-related work, Xu Ji has gained an in-depth understanding of the industry’s development. Over the years, he has become highly familiar with the issues facing China’s database industry and is passionate about contributing to its growth. To this end, he actively encourages sharing the experiences accumulated in the database field, writing books, creating WeChat accounts, and giving talks to spread knowledge about databases.
“I enjoy sharing. I hope everyone can work together to make the ecosystem better. When I published my books, many people told me not to because they thought the techniques and methods were the DBA’s bread and butter, and sharing them would make everyone else capable. But in the end, I still insisted on publishing,” Xu Ji said.
In 2017, after founding Nanjing Jishi Data Company, Xu Ji decided to launch an intelligent diagnostic tool for databases, hoping to make his experience available to a wider audience. This tool, called D-Smart, combines the experience of database experts into a knowledge map, automatically analyzing and diagnosing data collected from database operations such as logs and metrics, providing alerts when issues arise.
“D-Smart serves different types of DBAs in different ways. For ordinary DBAs, it helps identify common database issues and perform basic diagnostics and alerts. For advanced experts, it helps compile information and facilitate their diagnosis,” Xu Ji said.
Since D-Smart aims to support all mainstream databases in the market, Xu Ji has worked with many domestic databases over the years. He has also noticed their shortcomings. Many domestic databases face the awkward situation where small issues are ignored, but large problems become impossible to solve even with DBA intervention.
“Database observability is a key indicator of a database’s maturity. Oracle provides a vast array of metrics, such as over 2800 different wait events. Many domestic databases, however, don’t even provide basic metrics, which makes them less DBA-friendly,” Xu Ji said.
D-Smart is a deep maintenance tool that relies on rich metrics to make accurate judgments. However, domestic databases, especially newer distributed ones, often lack sufficient data and documentation, which limits the effectiveness of tools like D-Smart.
However, Xu Ji has had a good impression of OceanBase. It was the first domestic distributed database integrated with D-Smart,
demonstrating notable progress in observability compared to other similar databases.
DBA Requires Experience
As a database expert, Xu Ji believes being a DBA is a profession that requires accumulated experience over time. It’s a job that can be done well into old age and offers great job satisfaction.
“Being a DBA requires time and experience. It’s a position that blends theory with practice. If you’ve been doing it for three to five years, you’re just getting started. Some people might think they’re great by then and stop learning, but they’re really just entering a new stage,” Xu Ji said.
He believes the key to motivating oneself to keep learning is interest in databases.
“You need to be interested in databases. If you’re not, you might find yourself tired of the job as you age. Only if you’re interested can you stick with it long-term. DBAs become more valuable as they get older, because their experience enables them to quickly pinpoint and solve problems,” Xu Ji explained.
Furthermore, being a DBA requires strong learning abilities.
“You must be someone who enjoys learning, otherwise, it’ll be hard to stick to it. Simply reading documents and materials can be a challenge, let alone obtaining DBA-related certifications,” Xu Ji emphasized.
He still maintains a habit of learning, subscribing to many membership services and regularly reading books and materials to keep up with the field.
Lastly, he believes DBAs need to be calm and composed.
“Clumsy mistakes can lead to big issues, and once a database has a problem, it’s a big problem. From a long-term perspective, database maintenance is becoming more intelligent, but true high-end database experts can never be replaced by machines,” Xu Ji said.
Visit the https://payhip.com/OracleeBookSoftwareShop website to get an e-book about Oracle internal storage.
This content originally appeared on DEV Community and was authored by XINZ