This content originally appeared on DEV Community and was authored by XINZ
How can a DBA quickly locate system faults? How can performance be effectively optimized? How should one approach technology selection and build a robust technical system?
Today, we introduce a seasoned database expert and operations professional with over a decade of industry experience — the Operations Center Manager of the Information Technology Department at a major bank. He has been responsible for maintaining hosts, middleware, and databases; participated in the development of the bank’s core and payment systems; and led the construction of the operations framework, emergency disaster recovery system, and monitoring infrastructure. We hope his experience provides valuable insights for your own work.
In 2001, Li Jianming knew nothing about this issue when he was filling out his college application preferences. Unlike many who picked computing because they “liked playing with computers” or “found programming fun,” Li had never encountered computers in high school — at that time, computers had not yet become ubiquitous in campuses. He had only heard that learning computer science could lead to a good job after graduation.
Thus Li Jianming became among China’s first undergraduates in information security technology. During his university years he also moonlighted developing office software, forums, and blogs, and for the first time experienced the joy of programming:
“I really enjoyed the process of creating something from nothing, step by step building a product.”
Because the discipline of information security is broad, and Li wanted to build a strong foundation in computing, he decided to go deeper into engineering and strengthen his code-level foundations. So when applying for postgraduate studies, he chose the specialization of system analysis and integration. Li thinks an important factor in choosing a major is: “Is this path wide or narrow in the future? In plain terms, when choosing a direction for your career later, will there be large market demand or small? You should also consider your interests — if you pick something you are interested in, your path may go more smoothly.”
This choice planted a seed for Li Jianming to later enter the database field and become a DBA.
Growing as a DBA: Rapid Fault Diagnosis, Understanding System Mechanics
In 2008, after completing his postgraduate degree, Li Jianming joined a rural credit union (credit cooperative) during a period of rapid digital reform. He participated in maintaining and developing core systems and payment systems. As early as 2004, the State Council had called for deepening reforms of rural credit cooperatives, and in response, this credit union set up a Technology & Settlement Center in 2005 and built its information system. These efforts brought more convenient and faster financial services to customers. But ever-increasing transaction volumes posed challenges, making system stability more difficult to guarantee.
“Speed” as a Major Challenge
The Technology & Settlement Center where Li worked was responsible for building the credit union’s information systems. From his entry, he was in charge of both maintaining the systems and optimizing them, and in his spare time he self-studied DBA‑related knowledge. As a system maintainer, Li needed to locate faults quickly when the system erred and restore service fast. Achieving that “speed” was very challenging for a newcomer in tech.
Li’s approach was to deeply understand the system’s operational mechanics and execution flows. When an error occurred, he could quickly identify which accounts were wrong, and in what circumstances this would arise.
In general, operations personnel often feel that application systems always lack proper documentation. Developers may not fully document. Hence, maintenance engineers need to dig deeper. For example, understand the system architecture and business architecture and their implementation path; knowing table structures gives you a grasp of business structure; knowing OS commands helps you understand system layout; and one must further study system components, such as database table designs.
By continuously diving into layers of the system and exploring upward, Li gradually mapped out the running mechanism and execution flow, and through repeated hypothesis and validation he mastered the truth. After many “experiments,” he naturally acquired the ability to quickly locate and fix faults. Beyond that, Li spent seven or eight months reading the main source code of the system. That period of deep study and self-learning not only equipped him to rapidly locate issues in error states, but also greatly improved his technical ability and code quality.
In 2010, after two years of strong performance, Li Jianming was formally promoted to system administrator and database administrator (DBA), responsible for system stability and efficiency, and managing and maintaining the database, operating system, middleware, applications, etc. “At that time we had two system admins — whenever technical faults occurred, both of us had to step in to handle them.” The years he had spent exploring system internals made him intimately familiar with the entire system, allowing him to handle new tasks with confidence.
Experimentation, Mistakes, Learning
These problems might not be hard for a seasoned DBA — but for a DBA in growth, one must keep exploring, making mistakes, and learning through practice.
For instance, when a database connection pool fails, a less experienced DBA might assume “not enough connections” and simply create more. But when the connection pool is saturated, the problem remains. Because a full connection pool is only a symptom. The root cause must be analyzed. In some cases the workload indeed increased, but most of the time the pool is blocked because new business is inefficient or connections are not released in time. Simply increasing connections postpones the blockage, but doesn’t solve the core issue.
Another example: when a long transaction causes service interruption, a DBA unfamiliar with principles might just wait for rollback. But with experience, one learns that by increasing the transaction log files, one can ensure the rollback completes while the service continues to run.
“My work as system admin helped me gain deeper understanding of software systems. From theory to practice, I accumulated rich experience in fault handling and performance optimization, boosting my technical confidence,” Li says. He also summarized his experience in ensuring system stability, diagnosing faults, and optimizing performance.
As a DBA, ensuring stability is paramount. Li believes three tasks are essential:
Prevention: Monitor network, storage, compute resources and the underlying infrastructure. Ensure the foundational framework stays relatively consistent.
During normal operations: Observe the system from a business perspective to detect anomalies or “sub‑healthy” states. Build observability, warnings, emergency measures, disaster recovery, and standardized operational management.
Emergency response: Use the system architecture to localize faults, combined with monitoring tools to assist diagnosis. For fault localization and performance tuning, he suggests three angles:
a) Confirm system architecture and business structure — diagnose from a holistic view.
b) Watch various metrics: CPU, memory, I/O, load, business volume, success rate, response times.
c) Under stress, isolate performance of each module (front to back), or inspect summary and detailed activity logs: which functions were called, which SQLs executed, what each thread is doing, and even network packet response times or latency.
He also emphasizes that to avoid module owners shifting blame, there should be a technically integrated team where everyone can oversee the whole and dive in to resolve issues in emergencies.
Seasoned DBA: Choosing the Best, Not the Most Expensive Database
Over ten years at the credit union, Li matured from an inexperienced graduate into an experienced sysadmin and senior engineer. The credit union’s systems scaled from supporting millions of transactions to tens of millions. If that institution was the foundation of Li’s career, the bank he later joined became a high-level proving ground for his skills and experience.
In 2018, Li joined a bank in Yunnan, responsible for managing the operations team and overseeing the architecture of operations, disaster recovery, and monitoring systems. In 2021, he participated in a task seemingly distant from DBA work: database selection.
“Choosing a database without business context is reckless. To make a good tech choice, first you must know which databases are viable options in the market.” Li believes an excellent database product should be stable and simple like Informix, while offering rich built-in system tables like Oracle, and possessing high performance. It should meet ACID (Atomicity, Consistency, Isolation, Durability) requirements, and support application transparency so the DBA can see what tools are used, how much CPU is consumed, how much time a query took, and what operations were done.
Second, choose the “optimal solution” for your business. In the banking industry he worked in, most banks use Oracle because of its mature documentation, tooling, and ecosystem. These make it easier to find solutions or hire Oracle-capable developers. But as data volumes grow, Oracle’s data handling becomes a bottleneck. In multi-active (geo-distributed) scenarios, Oracle’s global cache mechanism can degrade performance when customer A’s data resides in data center A and customer B’s in center B — cross-access slows things down. Also, hardware and software costs are rising. Swapping out Oracle is not easy, and his bank was in a dilemma when selecting alternatives.
What banks truly need is a database with better performance at lower cost, one that can flexibly scale out and in — when load is low, reduce nodes; when traffic peaks, scale up. Compared to the traditional centralized Oracle, distributed databases looked like a promising alternative.
From a system perspective, Li compared top distributed databases and believed architecture should be decided by the database itself — not the upper layer or DBA always worrying about data placement. That criterion excluded many options. OceanBase’s unified architecture gave it an advantage. In terms of scaling, there are database-level, table-level, and row-level elasticity: database-level scaling is too coarse, row-level is not currently necessary for their transaction volumes, so a table-level scaling solution like OceanBase was appealing. Moreover, OceanBase’s “three-region, five-center” deployment ensures no data loss. From both performance and proven cases, OceanBase stood out: it has supported 10 years of Double 11 loads, and has been validated by institutions like Alipay and MYbank. In performance tests, it reached 8,000 TPS (transactions per second).
Because the bank in Yunnan used a traditional core system augmented with a distributed database, after careful evaluation they decided that OceanBase fit best. Having participated in the selection, Li laid out four key lessons:
The technology and product strength of domestic (Chinese) databases are improving; they should be part of consideration.
Facing ever-growing data volumes, distributed databases are a good choice.
When choosing a distributed database, consider its ACID guarantees.
Consider the cost of application modifications and compatibility, as well as performance, scale, and hardware compatibility.
Seven Growth Recommendations for DBAs
In the interview’s final segment, when asked what qualities or skills an excellent DBA should possess, Li drew on his ten-plus years of professional experience and offered seven suggestions:
Solid theoretical foundation in databases. For example, knowledge about database systems, core database concepts, and principles of distributed databases. Theory provides high-level guidance to practical work.
Familiarity with software development fundamentals and architecture. A DBA may not need to write perfect code, but if they don’t understand how code is written, load balancing, how to connect to databases, or common frameworks, then when troubleshooting they might only say “I don’t think the database is the issue” — and they won’t be able to ensure system stability from a holistic perspective.
Proficiency with operating systems and performance tuning. Databases ultimately run on operating systems. You develop OS familiarity via daily work, and for tuning, you should study official documentation, understand parameter meanings and the effects of changes, and practice hands-on regularly.
Strong experience in database operations and maintenance. Especially battle-tested in high concurrency and large data volumes. Operational fluency is built through volume of experience. Whether you encounter high concurrency depends on your company’s business. Many small Oracle deployments run fine with default settings, giving the DBA little challenge. That limits experience accumulation in demanding scenarios.
Master the hardest theories. Among many technical fields, first learn one deeply before branching horizontally. If you know many things at a shallow level, you’ll struggle to reach high proficiency in any.
Maintain curiosity and commit to lifelong learning. As a tech person, you can read technical books to learn IT theory; use platforms like Geek Time for hands-on system experience; read vendor documentation for domain knowledge; browse CSDN for difficult issues; use “得到” (Dedao) app for discipline and general knowledge; read papers for research-level and theoretical content.
Cultivate reverse thinking and structured reasoning. Break thinking inertia and imagine multiple possibilities — especially think toward extremes. Keep asking questions, overturn your own hypotheses and validate new ones.
If he could give advice to his younger self, Li Jianming said:
“Life is finite but knowledge is infinite. Be willing to relinquish and choose the direction you are most interested in or best at. Put in hard work to master your ‘killer skill.’ Also learn cross-disciplinary fundamentals to broaden your knowledge base, enhance multidimensional thinking ability, and be diligent in thinking and hands‑on experimentation.”
Visit https://payhip.com/OracleeBookSoftwareShop website to get an e-book about Oracle internal storage.
This content originally appeared on DEV Community and was authored by XINZ