It's a feeling I've come to associate with a vivid, recurring image in my mind – that of a giant arcade coin pusher machine. You know the ones: where a shelf full of coins slowly inches forward, pushed by a mechanical arm, until some of them tumble over the edge, creating a satisfying cascade and hopefully, a jackpot.

In my mental metaphor, however, the coins aren't inanimate currency; they're people. We are all on that platform, and the relentless march of time, technology, and societal advancement is the mechanical arm, pushing us forward.

As we get older, the world seems to accelerate its pace, widening the gap between what we know and what is becoming. Every day, the lexicon changes, new paradigms emerge, and the familiar ground shifts beneath our feet. We know less of what is happening, understand even less of what the changes truly mean, yet these changes fully affect us. The once-stable landscape of our careers, our social interactions, even our understanding of basic functionality, begins to erode. We are getting closer and closer to the edge, where the familiar drops away into the unknown.

In this human coin pusher, the front of the platform is often occupied by the older generations, those who have seen countless waves of change, and are now perhaps weary from the constant adaptation. They are the ones feeling the most immediate and profound force of the push, their accumulated knowledge sometimes deemed less relevant by the shiny new coins appearing further back on the platform. Their experiences, while rich and invaluable, can feel devalued in a world obsessed with the next big thing.

But it's not just age that determines your position. Sadly, the cruel logic of the coin pusher also means that some younger people find themselves pushed to the front prematurely. Perhaps it's an industry that became obsolete, a skill set no longer in demand, or simply an unfortunate turn of economic events. They, too, experience the jarring sensation of being rendered obsolete, even as their peers further back on the platform seem to be thriving in the new currents. The front edge of the coin pusher isn't exclusively for the old; it's for anyone caught in the wrong place at the wrong time, anyone who hasn't managed to adapt or find a new foothold.

The AI conference hammered this metaphor home. The sheer velocity of progress in artificial intelligence felt like the mechanical arm speeding up, its push becoming more insistent. New tools, new concepts, new ethical dilemmas – each presentation was another nudge forward, another reminder that the landscape of knowledge is being redefined at an unprecedented rate. My initial thought was one of frantic catch-up, a desperate attempt to grasp every new acronym and understand every algorithm. But then came the deeper realization: it's not just about keeping pace; it's about acknowledging the possibility of being pushed out, regardless of how hard you try.

This isn't a lament against progress, nor a call to halt the relentless march of innovation. It's an observation, a shared human experience in an ever-evolving world, nothing new. The coin pusher of life is a constant. The platforms will always move, new coins will always be introduced, and some will always reach the edge. Perhaps the true challenge isn't to avoid the edge, but to enjoy the mere existence of the game. An idea that has always felt comfortable to me, Absurdism, the philosophy championed by Albert Camus. Camus argued that there is a fundamental conflict between the human longing for order and meaning and the "silent," chaotic indifference of the universe.

Watching the coin pusher of society, we realize the machine doesn't care about our experience, our history, or our efforts to stay on the ledge. The push is indifferent. To look at the rapid, confusing advancements of the world and feel "pushed out" is to come face-to-face with the Absurd.

So, what do we do when we feel that inexorable nudge toward the edge?

Camus suggested that we shouldn't despair or retreat into false hope. Instead, we should live in defiant rebellion. We recognize that the machine will eventually push us over, but we continue to think, to create, and to observe the game with a wink. We find joy in the silver reflection of the coins and the hum of the motor, even as we move toward the drop.

Perhaps the true challenge isn't to avoid the edge, but to walk toward it with our eyes wide open. We may be getting pushed out of the world’s current "wave," but there is a strange, quiet freedom in acknowledging the ledge. As Camus famously concluded in The Myth of Sisyphus, as his protagonist watched his rock roll back down the hill for the thousandth time: "One must imagine Sisyphus happy." Even on the edge of the coin pusher, we can be happy—not because the machine stopped, but because it no longer has the power to surprise us.

Today, we are introducing a new player to the team: AI Code Reviewers (like Atlassian Rovo).

This doesn't mean we throw away our old process. On the contrary, our original structure is more relevant than ever. The AI simply changes who validates our work.

Here is the updated guide on how to define a task effectively, ensuring it satisfies both the human developer and the automated agent.

The Core Philosophy: "Can I do this without programming?"

Before we even open a ticket, the "Analyst-Reviewer Conversation" must happen. The most important question remains: "Can I do this without programming?"

• Yes: Use existing settings, parameters, maps, or global variables. Do not write code if configuration will suffice.

• No: Then the goal changes. Can I build this feature in a way that allows us to do it without programming next time?

I consider this question a structural turning point in how we design features. Treating every client request as a potential configuration problem rather than a coding problem forces us to think in terms of capabilities, not patches.

Instead of implementing one-off logic, we ask whether the requirement can be absorbed into the system as a reusable mechanism. When the answer is yes, the application evolves by extending its own degrees of freedom: what previously required development becomes a matter of configuration. This is how software transitions from a collection of special cases into an adaptive platform. This is what coreBOS and EvolutivoFW are!

Consistently applying this principle changes the trajectory of the product. It encourages us to build infrastructure that preserves existing behavior while expanding what the system can express. Over time, this reduces future development effort, increases operational flexibility, and allows the application to respond to new business demands without continuous structural change.

In short, this question reframes feature requests as opportunities to increase the system’s long-term adaptability rather than merely satisfying the next requirement.

But this question raises another concern: “How do I know if it can be done without programming?”

If you are unaware that a global variable or a configuration setting already exists, you will naturally default to writing code, which is exactly what we want to avoid. Overcoming this "unknown unknown" requires a culture of Shared Knowledge. You cannot operate in a silo. You must feel empowered to ask—and specifically, to ask those teammates who have a history of sharing and mentoring.

This is where Training and Reading become your professional responsibility. You must actively study the system to understand the tools available to you. But the most critical piece of this puzzle is Documentation. You must write the documentation you wish you had found. Every time you solve a problem via configuration rather than code, document it. This acts as a "pay it forward" mechanism: it ensures the next person finds the answer in our knowledge base rather than reinventing the wheel in the codebase.

So, depending on the answer to the question of whether we can create new infrastructure or not, we arrive at one of two situations:

• Yes: Ask permission and validation because this path will take more time than a direct hack solution. It is time you will save next time this requirement is needed

• No: Develop the easiest compatible solution

If the answer is "We must code", the discussion that happened here holds the information we need to continue and create the ticket.

Title and Description: The "Human" Layer

The top half of the ticket is for the humans. It provides the "Why" and the context that an AI cannot fully grasp.

• Title: This must be a concise summary. Ideally, this text should be clean enough to serve as the Commit Message later.

• Description: This is the detailed explanation. It should include steps, links to designs, screenshots, and the business context.

  • 💡

    Note: You can write this in your team's native language. The AI doesn't strictly need to "understand" the business benefit (like reducing churn), but your team does.

Validation: The "AI" Layer

This is the most critical update to our process. In my original guidelines, VALIDATION was a checklist for the human tester.

Now, the Validation section is a prompt for the AI.

When using tools like Atlassian Rovo or GitHub Copilot for PR reviews, they look for specific instructions to verify the code against. To make this work, the Validation section must follow strict "Machine-Readable" rules.

How to write the Validation section for AI:

1. Use the "Magic Words": The AI scans for a specific header. You must label this section "Acceptance Criteria" or "Definition of Done" (Case sensitive).

   1. 💡

      If you are in a rush, Rovo also recognizes the standard shorthands AC, ACs, or DoD. I recommend you use your system’s templating engine to write this for you.

2. With current tooling, Language Must Be English: Even if the rest of the ticket is in Spanish, the Validation criteria must be in English for the current generation of AI agents to verify it against the code.

   1. ⚠

      This is what I have read, which really surprises me, though, and probably will not be true if you are reading this in the (near) future.

3. Be Binary (Pass/Fail): The AI compares the text to the code.

   • Bad: "Check that the API works well." (Subjective).

   • Good: "API Endpoint POST /v1/tickets/ exists and accepts parameter id." (Measurable).

Example: The "Translation" Strategy

We separate the Context (Description) from the Verification (Validation).

Atlassian has some good recommendations

• Use AI to turn your brainstormed thoughts into clear, structured work item descriptions.

• Use clear, unambiguous language.

• Keep each criterion statement short and focused on one specific thing.

• Break large epics into smaller stories with clear “done” conditions.

One last comment that is worth saying. Even in the time of AI all over the place, the rule of Garbage-In → Garbage-Out is still valid. AI does not reduce ambiguity; it amplifies it. If the criteria are vague, the review will be meaningless.

The "60% Stop and Ask Rule" is Now Automated

My favorite rule has always been: "At 60% of effort, stop and ask: Am I finished?"

The logic is that the remaining 40% of the effort is testing, documentation, and review. If you haven't finished the core logic by the 60% mark, you are off track.

The AI now enforces this rule for us.

When you open a Pull Request (usually around that 60-70% mark), the AI Code Reviewer scans your code against the Validation section. It instantly tells you:

• ✅ Criteria Met

• ❌ Criteria Missing

• ⚠️ Manual Check Needed

If the AI marks a criterion as missing, the answer to "Am I finished?" is objectively No. You don't need a senior reviewer to tell you that you forgot the database migration; the system catches it immediately.

Remember to keep stakeholders informed. If the dates are slipping, notify your peers. Keep the conversation moving!

Effort & Priority

Finally, remember that Effort is super important.

• Dates: We may hate deadlines, but we must notify the team if dates slip. Keep the conversation moving!

• Priority: always urgent

• Tags

• Stakeholders: informer, validator, …

An example

Title: Implementation of Call to Action Flag

Description (Context): To optimize marketing campaigns, we need to centralize the call-to-action status. Currently, there is a blind-spot whereas we do not know if the user has done the action or not.

The goal is to use our central application as a "Single Source of Truth" from where we can confidently determine if a contact should be included or not in an email marketing campaign.

Acceptance Criteria (Rovo will scan this section. Must be in English.)

1. Database: Add a boolean column cta_done to the Contact table.

2. API: Implement endpoint POST /v1/contacts/{id}/cta-done

3. Logic: The endpoint must be accessible only by a user with a valid API token.

4. Logic: The endpoint must accept a JSON payload with contact_id.

5. Logic: When the endpoint is called, update cta_done to true.

Business Goals (Manual Verification) (kept separate so Rovo doesn't flag them as "Missing Code")

• ⚠️ Churn: Verify that users with this flag stop receiving emails.

• ⚠️ Conversion: Verify that marketing spend is focused only on users with a 0€ balance.

Priority: Medium

Dates: 2026-02-20

Tags: API, Marketing

Effort: 1 day

Summary

The goal of a task definition hasn't changed: we want to avoid wasted effort.

• The Description ensures the humans know why we are building it.

• The Validation section ensures the AI can verify what we built.

By being strict with our Validation criteria—using English and specific headers—we turn our ticketing system into an automated quality assurance engine.

However, a quieter, more humbling realization is hidden in that definition.

If we shifted the goalposts and defined AGI as the average of human intelligence, the "intelligence explosion" wouldn't be a future event — it would be a retrospective one.

When we look at the "average" human baseline, we aren't looking at the ability to solve quantum equations or write symphonies. We are looking at a baseline that often struggles with media literacy, falls for basic phishing scams, and engages in circular logic on internet forums (without getting into what we are doing to the planet and with our political system 🤦‍♂️). By that metric, Large Language Models (LLMs) didn't just meet the average; they soared past it somewhere between GPT-3 and GPT-4.

The humor in this observation masks a technical truth: AI doesn’t need to be "God-like" to be transformative; it only needs to be slightly more competent than the median human at a specific task. We keep moving the goalpost toward the "Sum of Humanity" because if we admitted we’d already reached the "Average," we’d have to face the fact that our machines are already more "human" than we care to admit.

We aren't just building a mirror of our best selves; we are building a tool that proves the "average" was never as high as we thought it was.

Stay calm. We aren’t being replaced by superintelligence yet; we’re just being joined by a very fast version of ourselves."

I searched for tools to extract statistical information from a git repository, but found that most were designed for version control, not analysis. I eventually ran into gitstats, which does a great job at extracting metrics but generates a static snapshot. I showed the HTML report to my client, and the inevitable happened: he started asking, "Can I filter this by month?", "Can I restrict that to the backend team?", "Can I see only the refactors?"

It was the normal human process: the moment you start seeing data, you want to ask it questions. But static reports can't answer new questions.

I realized that hacking filters into existing tools wasn't enough, but urgency reigns, so I added some filtering options to gitstats and gave it to my client to cover the momentary needs. I was going to leave it there, but the ETL/BI engineer inside me was hooked. If we wanted real answers—like "Who was the most productive author of 2024?" or "Which files have the highest churn rate?"—we didn't need a report generator; we needed a data warehouse.

Thus was born GitStatsDB.

Today, I am releasing gitstatdb, an open-source ETL (Extract, Transform, Load) tool that turns your Git history into a structured MySQL database ready for dynamic reporting

The Concept: Your Code History as Data

The philosophy behind gitstatdb is simple: standard SQL is more powerful for reporting than Git commands. By extracting repository metadata and loading it into a relational database, we unlock the full power of BI tools like Metabase, Superset, or Tableau.

Unlike simple commit counters, gitstatdb captures the deep context of your project:

• Complete History: Commits, normalized authors, and committers.

• Branch Evolution: Tracks all branches, including those that have been deleted, ensuring historical accuracy.

• File Forensics: Tracks every file change (insertions, deletions, modifications) and file renames.

• Merge Relationships: Automatically detects source and target branches for merges, making it easier to visualize workflow efficiency.

Under the Hood

The database schema is designed for performance and detailed analysis. It features normalized tables for author, repository, and branch, linked via a central commit table.

We also include pre-calculated statistics tables (repository_statistics, branch_statistics, author_repository_statistics). This means that when you connect a dashboard tool, it doesn't have to crunch millions of rows in real-time—the heavy lifting is already done.

Incremental Updates

One of the biggest challenges with Git analytics is performance. gitstatdb supports incremental updates. After the initial import, you can run the tool daily (recommended actually); it detects new commits and processes only what has changed. It even detects when local branches have been deleted and marks them accordingly in the database.

Visualizing the Data

Once your data is in MySQL, the magic happens. You can connect tools like Metabase to visualize your repository's heartbeat.

I have created a set of advanced dashboards that track:

• Authors of the Month: Ranked by impact and consistency (not just commit counts).

• Code Churn: Identifying "hotspots" in your codebase that are frequently rewritten.

• Project Velocity: Visualizing merge rates and active days.

Watch this video to see how we use Metabase to explore a repository's history:

Installation & Usage

Getting started is straightforward. You can install it directly as a Python package:

# 1. Install
pip install -e .

# 2. Configure your database in a .env file
echo "DB_NAME=gitstatdb" > .env
# ... add user/pass ...

# 3. Run the ETL
gitstatdb /path/to/your/repository

For specific analysis, you can even force the recalculation of statistics for specific branches or the whole repo via the command line.

Commercial Reporting

While gitstatdb is open source (MIT License) and free to use, building the right SQL queries for advanced dashboards can be tricky.

The project includes a reporting directory with setup instructions for Metabase. However, the advanced template packs, complex SQL reports (like the "Authors of the Year" logic), and specific Metabase/Superset configurations are available as On-Demand Services.

If you want to skip the setup and jump straight to insights, you can contact me for the premium dashboard pack, which includes:

• Support for setting up the tool and the necessary crons.

• Pre-configured Metabase dashboards.

• Complex SQL views for Churn and Author ranking.

• Support for setting up Superset (others?) visualizations.

Get the Code

The project is hosted on GitHub. Give it a star and start treating your code history like the valuable dataset it is.

👉 GitHub - joebordes/gitstatdb

For some reason, I didn’t receive any notification about the event this year. I casually remembered about it at the end of September and went to see if it was still happening. Happily, I found that it was, and they had decided to return to the t-shirts and tree planting. This may seem stupid and unnecessary, but it made me happy and motivated me to participate once again.

My first surprise was with the registration information. They asked for more information than ever, which made me uncomfortable and suspicious of it being just another marketing scheme.

The next surprise was that they increased the number of commits to 6. Not that I think that is wrong, but it caught my attention.

In any case, this year found my mind actively looking for a reason to dedicate time to the MCP protocol. I had been reading and listening to talks about it for some weeks at this point, and I saw a clear connection with Evolutivo.fw (coreBOS), which was the perfect excuse to finally sit down and dive into it.

My Contributions

1. Translating TiddlyWiki to Spanish

My first pull request is the traditional TiddlyWiki ES translation. Thanks Jeremy

2. Helping TCPDF

Then I got a follow-up question in an issue report I made to TCPDF. They aren’t participating in Hacktoberfest, but I decided to help anyway. Even though they aren’t part of Hacktoberfest, contributing still feels in the same spirit.

3. Building the Evolutivo.fw MCP Integration

Next, I started the coreBOS (Evolutivo.fw) MCP project. Once I learned the high-level concept of MCP, I saw how it could help working with the Evolutivo application. If we could ask questions and work with the application from an intelligent chat environment, it could be a decisive step forward for the future of the project.

So I started studying the MCP protocol and implemented a complete integration for CRUD operations and more. You can read all about it in this blog post. For the Hacktoberfest participation, this project alone contributed the 6 commits I needed and many more.

4. Updating Node-RED Language Files

   Next, I updated the Node-RED language files, like I did last year. They accepted the pull request a few days later and were kind enough to add the hacktoberfest-accepted label so it counted towards my participation. Thanks!

5. Translating Mautic to Spanish

Finally, I translated some Mautic strings, a little over 200, to ES and created a PR with some syntax and grammar errors I found along the way.

Hacktoberfest Participation

As a conclusion, I can say that small contributions here and there matter and that this is another great reason to dedicate time to new projects.

So, another year where I can still say, with pride, that I have participated in all the challenges of Hacktoberfest that have been held. Looking forward to seeing what happens next year!

I insisted on staying away from it, that is, until this past week. The inevitable happened, and I was forced to face it head-on. A reseller informed us that his applications were working way too slowly. I had no idea where the problem could be, but I suspected that it lay somewhere deep within the database or (hopefully) in the server itself. I had no choice but to learn.

What I discovered was that it was far easier than I imagined. And, to be honest, a lot of that is thanks to generative AI, which has changed everything, here too.

Let's dive in.

The Tools

My newfound database optimization arsenal consists of just a few key tools. Together, they form a workflow that takes you from a vague sense of a problem to a precise, actionable solution.

• mysqltuner

• etckeeper

• MySQL mysqld.cnf

• Percona pt-query-digest

• Your favorite generative AI

The Workflow

1. The Quick Audit: mysqltuner

mysqltuner is a Perl script that acts as your friendly MySQL performance auditor. You run it from the command line, and it connects to your MySQL instance to give you a quick report card. It analyzes everything from your buffer settings and thread cache hits to your key-read efficiency and a host of other metrics. It's not a diagnostic tool for a specific query, but rather a "general health check" that gives you a high-level overview of where your configuration is weak.

In the general case of Evolutivo, and in particular in the case I was debugging this week, the (edited) output at the start was

[!!] /var/log/mysql/error.log contains 48 warning(s).
[OK] /var/log/mysql/error.log does not contain any error.
[OK] Maximum reached memory usage: 26.8G (45.55% of installed RAM)
[OK] Maximum possible memory usage: 28.1G (47.72% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (22/36M)
[!!] Highest connection usage: 89%  (135/151)
[OK] Aborted connections: 0.00%  (4/176256)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[OK] Sorts requiring temporary tables: 0% (193 temp sorts / 1M sorts)
[!!] Joins performed without indexes: 50789
[OK] Temporary tables created on disk: 0% (121 on disk / 1M total)
[OK] Thread cache hit rate: 99% (1K created / 176K connections)
[!!] Table cache hit rate: 0% (2K open / 1M opened)
[OK] Open file limit used: 2% (218/10K)
[OK] Table locks acquired immediately: 99% (712K immediate / 713K locks)
[!!] Binlog cache memory access: 83.85% (225656 Memory / 269115 Total)
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 16.0G/4.7G
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 2.0G * 2/16.0G should be equal to 25%
[OK] InnoDB buffer pool instances: 16
[--] Number of InnoDB Buffer Pool Chunk : 128 for 16 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.99% (3807017853 hits/ 3807246008 total)
[!!] InnoDB Write Log efficiency: 60.97% (2079430 hits/ 3410771 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1331341 writes)
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mysql/error.log file
    Reduce or eliminate persistent connections to reduce connection usage
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Adjust your join queries to always utilize indexes
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: https://bit.ly/1mi7c4C
    Read this before increasing for MariaDB https://mariadb.com/kb/en/library/optimizing-table_open_cache/
    This is MyISAM only table_cache scalability problem, InnoDB not affected.
    See more details here: https://bugs.mysql.com/bug.php?id=49177
    This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
    Beware that open_files_limit (10000) variable 
    should be greater than table_open_cache (2000)
    Increase binlog_cache_size (Actual value: 32768)
Variables to adjust:
    max_connections (> 151)
    wait_timeout (< 28800)
    interactive_timeout (< 28800)
    join_buffer_size (> 80.0M, or always use indexes with JOINs)
    table_open_cache (> 2000)
    binlog_cache_size (16.0M)

Reading the procedures that we have implemented as a company in the past and the output above the settings that stand out are

• table_definition_cache: When a client establishes a connection to a MySQL server, the server maintains a cache of table definitions in memory so that it can quickly look up metadata about tables that are used in queries. The table_definition_cache variable controls the size of this cache. Evolutivo has a LOT of tables, so we need to increase this size to avoid having to go to the disk for the information.

• max_connections: in a server with many installations, even of other applications like WordPress, PrestaShop, Nextcloud, … which all use the MySQL server, this value has to be high

• innodb_redo_log_capacity: The innodb_redo_log_capacity setting controls the total size of the redo log files. A larger capacity allows MySQL to defer flushing changes from memory to disk, which can significantly improve write performance, especially during high-volume operations.

• skip-name-resolve: this avoids extra network calls, but be careful, some applications require this setting if not set to work with IP or subnets only.

After setting these variables (and some other work), I ended up with this output

[OK] Maximum reached memory usage: 17.3G (29.46% of installed RAM)
[OK] Maximum possible memory usage: 36.1G (61.29% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 2% (19K/735K)
[OK] Highest usage of available connections: 6% (16/251)
[!!] Aborted connections: 4.03%  (295/7328)
[OK] Sorts requiring temporary tables: 0% (10 temp sorts / 46K sorts)
[!!] Joins performed without indexes: 217
[OK] Temporary tables created on disk: 0% (1 on disk / 18K total)
[OK] Thread cache hit rate: 99% (58 created / 7K connections)
[OK] Table cache hit rate: 29% (4K open / 16K opened)
[OK] Open file limit used: 5% (520/10K)
[OK] Table locks acquired immediately: 99% (74K immediate / 74K locks)
[OK] Binlog cache memory access: 100.00% (12160 Memory / 12160 Total)
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 16.0G/4.7G
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 2.0G * 2/16.0G should be equal to 25%
[OK] InnoDB buffer pool instances: 16
[--] Number of InnoDB Buffer Pool Chunk : 128 for 16 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.90% (44294781 hits/ 44339899 total)
[!!] InnoDB Write Log efficiency: 73.46% (154640 hits/ 210498 total)
[OK] InnoDB log waits: 0.00% (0 waits / 55858 writes)
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mysql/error.log file
    Control error line(s) into /var/log/mysql/error.log file
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce or eliminate unclosed connections and network issues
    Adjust your join queries to always utilize indexes
Variables to adjust:
    join_buffer_size (> 80.0M, or always use indexes with JOINs)

A significant improvement!

But, before you do that, let’s see the other tools.

2. The Safety Net: etckeeper

This tool isn't directly related to MySQL performance, but it's a total lifesaver for any Linux server. etckeeper is a collection of scripts that hooks into your package manager (apt, dnf, etc.) to automatically commit changes to the /etc directory into a local Git repository.

In our case, we'll be modifying the core MySQL configuration file, mysqld.cnf. Having Git track and version every single change we make is of immeasurable value. If you make a mistake, you can simply roll back. This simple safety net removes a lot of the fear and hesitation from performing server-level changes.

3. The Configuration File: mysqld.cnf

This is the main configuration file for your MySQL server. It's typically located in /etc/mysql/mysql.cnf or /etc/my.cnf. mysqltuner will suggest several changes to this file, such as adjusting buffer sizes or increasing certain cache limits. You'll make these adjustments, save the file, and restart MySQL. Don't worry—etckeeper will be there to save you if anything goes wrong.

You will have to activate slow query analysis and let your users work normally for some days and repeat during a few weeks.

4. The Forensic Analysis: pt-query-digest

This is the most crucial tool in the entire workflow. pt-query-digest is a script from Percona Toolkit that analyzes your MySQL slow query log. It reads through millions of queries and then, like a skilled detective, presents you with a report that tells you exactly which queries are causing the most pain. It will identify the worst offenders based on total execution time, lock time, and number of rows examined. This report is the goldmine of information you need to move forward.

5. The Final, Easy Step: Generative AI

Now comes the fun part. You have the raw data from pt-query-digest, which, to the untrained eye, can still be a bit overwhelming, it looks like this:

# Query 11: 0.00 QPS, 0.00x concurrency, ID 0x37107AE09E34F1D59DD462CB636DEBDB at byte 1188119
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.00
# Time range: 2025-08-30T10:32:43 to 2025-08-30T13:26:10
# Attribute    pct   total     min     max     avg     95%  stddev  median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count          0      44
# Exec time      0      2s    25ms    69ms    34ms    42ms     9ms    31ms
# Lock time      0   103us     1us     9us     2us     5us     1us     1us
# Rows sent      0      48       1       2    1.09    1.96    0.28    0.99
# Rows examine   8   2.25M  52.28k  52.28k  52.28k  52.28k       0  52.28k
# Query size     0   2.32k      54      54      54      54       0      54
# String:
# Databases    xxxxx
# Hosts        localhost
# Users        xxxxx
# Query_time distribution
#   1us
#  10us
# 100us
#   1ms
#  10ms  ################################################################
# 100ms
#    1s
#  10s+
# Tables
#    SHOW TABLE STATUS FROM `xxxxx` LIKE 'vtiger_email_track'\G
#    SHOW CREATE TABLE `xxxxx`.`vtiger_email_track`\G
# EXPLAIN /*!50100 PARTITIONS*/
select * from vtiger_email_track where mailid='627858'\G

The final step is to feed that output into your favorite generative AI tool.

And then... you watch.

The AI will tell you exactly what is wrong with the query and, more importantly, how to fix it. It will explain that a missing index on a WHERE clause is causing a full table scan, or that a join condition is inefficient. It will then give you the precise ALTER TABLE commands needed to create the necessary indexes.

For the output above, it gave me this response

This pt-query-digest output shows a query that, while individually fast, is highly inefficient and contributes to overall database load due to frequent, repetitive full table scans.

Analysis

• Frequent Execution: The query runs 44 times, indicating it's part of a routine operation (e.g., loading a user's mailbox view).

• Excessive Rows Examined: The most critical issue is that each execution of the query examines an average of 52,280 rows to return only about one row. The total number of rows examined is a staggering 2.25 million. This is a classic symptom of a full table scan, where the database has to read every single row in the table to find the one that matches the WHERE clause.

• Execution Time: While the average execution time is only 34 milliseconds, a full table scan for a single row is still a waste of resources. The cumulative time across all 44 executions is 2 seconds, and with more users or a larger dataset, this problem would quickly escalate.

• The Culprit: The WHERE clause where mailid='627858' is the source of the problem. The mailid column is not indexed, forcing the database to perform an inefficient full table scan.

Recommendations

To solve this, you must add an index to the mailid column in the vtiger_email_track table. This is a simple but highly effective change.

1. Add an Index to mailid: Run the following SQL command to create an index on the mailid column.

    ALTER TABLE vtiger_email_track ADD INDEX `idx_mailid` (`mailid`);
   

   This will allow MySQL to use a highly efficient B-tree index to quickly locate the specific row(s) that match the mailid value, reducing the Rows examine count from tens of thousands to a handful.

By adding this index, you will dramatically reduce the load on your database, speed up this frequently run query, and improve the overall performance of the application that relies on it.

Sincerely, it is mindblowing to have extracted that response and conclusion from the output of pt-query-digest. In the past, this process would have required hours of research, EXPLAIN plan analysis, and a deep understanding of database internals. Now, with a handful of powerful, open-source tools, a dash of generative AI, and some local brain intelligence, you can go from being intimidated by MySQL to an optimization wizard in a single afternoon.

As a recap, the procedure is:

Go ahead, give it a try!

“The second is the first to lose.”

I’ve always hated that phrase. It’s the kind of idea that gets repeated until we stop questioning it. But I do. I question it deeply.

For me, it represents something broken — an outdated mindset that only one can win, that everything else is a waste. That if you’re not standing at the top, you may as well not have shown up. And that couldn’t be more wrong.

A few days ago, I watched the final of the EuroBasket Women’s Championship. Spain played against Belgium. Spain wasn’t supposed to be there. They weren’t the favorites. In fact, statistically, they weren’t even supposed to make it past the earlier rounds. But they fought. Hard. They played each match with grit, unity, and incredible heart — the kind of performance that defines sport at its best. They made it to the final.

They were on top for most of the match. It came down to the final seconds — a single basket. A heartbeat. That’s how they "lost". That is how basketball is. That is how our society is.

But they didn’t lose. They achieved something extraordinary. They overcame odds, inspired fans, and showed a level of excellence that deserves celebration. Not pity. Not a quiet mention between a weather forecast and the latest from Real Madrid CF.

And yet, the next day, barely anyone was there to greet them. Just a few family members. The headlines? “Sad defeat for Spain’s women’s basketball team.” Then silence. Then soccer (again).

This is where our values feel upside down.

In a society that claims to celebrate diversity, inclusion, and equity — especially for women, LGTBIQ+, or other underrepresented groups — we need to rethink what we reward and what we recognize. Not just in politics or policy, but in the narratives we build.

Sport isn't just about winning. It's about courage. It's about progress. It’s about doing what you love — and doing it with everything you've got, even when the scoreboard doesn't agree with you. That should be enough to command our attention and admiration.

Exactly like the professional who has found his passion and manages to support his family and the families of his coworkers, that professional who will never get invited to Stanford has a lot of merit that should be celebrated.

And while I’m here — a quick vote on another cultural bias: Why do we keep calling rain “bad weather”?

Seriously. I like it when it rains. It makes things grow. It quiets the world. Not every cloudy day is a problem to fix. Some of them are exactly what we need.

Maybe that’s the point.

We need to redefine what we call a win. What we call success. What we call good or bad.

Because sometimes, the team that “lost” the match won something much more meaningful.

And sometimes, a little rain is exactly what reminds us to look up. 🌧️

• The Understandably newsletter reminded me of the speech Steve Jobs gave at Stanford. Yes, that one, again. You can access it from the newsletter.

• Then, during a company meeting, the same theme was pitched again—but this time dressed up in the language of emotional connection, team-building, and community, rather than as... you know, a job.

• And just to really drive the point home, I randomly landed on an interview on TV while waiting for my wife, and there was Vanessa Martin passionately advocating for—yep—finding what you love.

By the third time, I wasn’t even surprised. I have even written and spoken about this in the past. I get the appeal. The idea is simple, almost seductive, powerful, as most simple ideas are. There’s an undeniable truth in the core message: Find what you love, pursue it with passion, and everything else—skill, career, happiness, money—will follow.

And here’s the thing: I believe in it. Mostly.

When you care about something deeply, you’ll put in the hours. You'll keep going even when it's hard. You'll get better, and eventually, you may even be considered a professional—someone who truly knows their craft. But that money and invitations to speak at Stanford or get interviewed on TV are where I disagree.

Because somewhere between passion and profession, the story gets… edited.

We’re told that money and tranquility follow automatically. That doing what you love will lead to financial security, peace of mind, and a meaningful life. That if you're not there yet, you just haven't tried hard enough or believed deeply enough.

Here’s where I call it out: that’s not the whole truth.

Yes, some people do find both peace of mind and serenity by following their passion. But it’s never just passion that gets them there. There’s always something more—access, timing, luck, connections, health, privilege, family support, genetic disposition, even geography. Things they don’t mention. Things they may not even be aware of themselves.

To make it sound like passion alone is the golden ticket is—let’s be honest—a bit misleading.

Because I also know people who are wildly passionate and incredibly skilled. People who are true professionals in their craft. And still, they’re struggling to pay their bills, supporting families, and dealing with anxiety about the future. They work hard, not just for joy, but out of necessity.

And yet, they're told they're missing something. That they haven’t truly followed their passion. That if they had, the money and ease would’ve come.

No.

That’s not a lack of passion. That’s the real world.

I wish we’d start telling the full story. That following your passion is worth it for the personal growth alone. That it might lead to success, but it might not. And that’s okay. That doing something meaningful is still meaningful, even when it’s hard. Even when it's underpaid. Even when it doesn’t lead to the TED Talk.

So yes—find what you love. But let’s stop pretending that that is all you need.

"Finding what you love may help you survive the grind. But it’s not a magic spell. It’s a seed. The rest still depends on rain, soil, timing… and a bit of luck."

We discussed this for a while until the network started failing. We couldn’t contact anybody, the internet, that network that was created to support nuclear war back in the Cold War was down in a whole country, maybe even more of the world, who knows? We had no means of getting news because everything runs on electricity now! We don’t even have a radio in the house, it is all digital.

At this point, I got up and ran to the window expecting to see bonfires and people chasing themselves around for food. What I saw was a beautiful day. A beautiful clear blue sky with some fluffy white clouds lazily floating around. Some people out for a walk and bicycle riding. The birds singing. I could even hear the waves in the ocean, probably because the constant whirl of my computer fans and everlasting construction noise was missing.

Definitely, I am watching too much television. I decided to grab a book and sat down for a read while the electricity returned. In the calm that only reading can bring to your brain and probably due to a constant lack of sleep, I inevitably fell into a pleasant nap. When I woke up, much to my surprise there was still no light. I looked out the window again to see that things had escalated. Now there were more people out in the street, talking, enjoying the day, and debating about the outage among other things. Interesting.

It was time for lunch, so I went to prepare a salad and some canned food. We had a nice lunch in the sun, discussing as we usually didn’t do, due to the fact that we couldn’t get captured by the cell phone which was still on the table. Then my son arrived, all hyped up about not being connected and what was he supposed to do now. I dryly answered: “take off your shoes, grab the leash, and take your dog for a walk on the beach. That still works, even without electricity.” To my surprise, that is exactly what he did after lunch.

I spent the afternoon doing household errands and DIY tasks. Then I went for a 4km walk to my mom’s house. I saw more people out on the street than usual, and, THEY WERE TALKING TO EACHOTHER. Incredible, people had no reason to stay stuck on the computer or have their heads on the phone while rudely ignoring the person next to them. They were enjoying being together. Even I stayed at my mother’s house more than usual.

On my arrival, I had a nice chat with the family. The houses around mine were doing the same, you could hear a nice rumor of chit-chat as the people interacted with the people around them. It reminded me of when I was much younger, people used to sit in the street and gossip for entertainment. Yes, I am that old.

I went to bed early and got a good night’s sleep for a change. Nice and needed.

When I woke up it was back to life as normal. No catastrophe, no cataclysmic post-apocalyptic event, nature didn’t even care at all that the light had gone out for 12 hours, and WhatsApp and Telegram were buzzing as usual.

I am very aware that there are many many edge cases here. People trapped in elevators, hospital emergencies, people who count on that electricity to breathe in their homes, a lot of income that was not generated, and others which I am not considering, but, maybe, just maybe, we should get into the habit of turning off the lights from time to time.

Photo by Dyu - Ha on Unsplash

Anyway, whatever. Here is my contribution to another year of open-source greatness.

My first pull request is the (now) traditional TiddlyWiki ES translation. Thanks Jeremy

During September I studied PHP Slim Framework. I found it to be a very interesting project. I like the simple but powerful approach and the ease with which you can extend and program in the framework. While I was studying the framework I immediately related it to coreBOS and thought of making a skeleton project for a “real” REST API. Thus was born my first idea for Hacktoberfest. I created the coreBOS REST API project, implemented the code for a REST API on top of the coreBOS web service API, and added the basic structure and functionality. Then, for the PR I added a MassDelete endpoint. This project defines how coreBOS officially supports a RESTful API.

At that moment I received an update about Node-RED and decided to update the ES language files. I did that and submitted the PR which got accepted a few days later but they aren’t participating in Hacktoberfest and didn’t attend my requests to join, so it didn’t count towards that challenge. It is more than good enough for me and, as I said at the start, doing four PRs in open-source projects a month is just second nature at this point…

Another few weeks went by as I was thinking of just letting the whole thing pass when I was teaching someone how to do coreBOS web service queries using the coreBOS Web Service Development tool and I ran into an idea that has been on my mind for years. I want to be able to do ctrl-enter in the query box to submit the query for execution. This functionality has been on my mind for years but it was so “insignificant” that I just never got around to dedicating some time to it: thus was born my next PR!

For my last PR I decided to update our Mattermost integration. I updated the Mattermost plugin to the latest Golang and Mattermost versions and then updated the Chat With Me extension to work with that plugin and the latest changes inside the coreBOS project. I did many commits during this update, among which I take special pride in the documentation, but I just picked a couple and marked them as hacktoberfest-accepted.

A total of 39 commits in 9 repositories during October

So, all in all, another year where I can still say, with pride, that I have participated in all the challenges of Hactoberfest that have been held. We’ll see what happens next year. Stay tuned!

From the manual page:

Pigz compresses using threads to make use of multiple processors and cores. The input is broken up into 128 KB chunks with each compressed in parallel. The individual check value for each chunk is also calculated in parallel.

So a gzip that uses all the processors in the server to do its work faster. Let's give that a try.

I look around the server and find a 14Gb SQL dump file of one of our databases. It's a perfect file for a compression test. So I compress and uncompress it with pigz and gzip.

time pigz cbcrm_pre_update_30052024.sql
time unpigz cbcrm_pre_update_30052024.sql.gz
time gzip cbcrm_pre_update_30052024.sql
time gunzip cbcrm_pre_update_30052024.sql.gz

The results:

The time for compression has a significant difference, while the uncompression is almost similar, I suppose due to the complexities of creating meaningful chunks of compressed data and the internals of the gzip format.

The htop output shows a clear difference in server resource usage:

These first two images are two different moments of the pigz execution:

where we clearly see all CPUs working together. The next two images are for the gzip execution.

where we see the lack of parallel computing. The unzip executions look similar.

In the Altlantic article referenced below and in the manual you will find some useful execution options and combinations.

A very practical use of programming and server knowledge! Kudos to the pigz team.

References

• pigz - Parallel gzip

• madler/pigz: A parallel implementation of gzip for modern multi-processor, multi-core machines.

• compress/expand files - Linux man page

• How to Install and Use Pigz to Compress File in Linux

• Image by vwalakte on Freepik

Node-RED defines itself as a programming tool for wiring together hardware devices, APIs, and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single click.

Internally, we seem to be pushing Jitsu these days, but mostly because we have our sites on the "mass ingestion of events" not because it fits this category. Mage has caught my attention but it seems overkill for a lot of the "simple" moving information around, that we normally need (much like Jitsu does).

This past week I had a task to read from an Apache Kafka queue and send an HTTP request to a Cumulocity endpoint, so I started searching for a tool where I could create a Cumulocity connector that would give me a nice environment to work in and some additional functionality.

After a while, I had reduced the list you can find below down to:

• StackStorm

• Prefect

• node-RED

• luigi

I already had made my decision to work with prefect. It produced the best sensation over the rest, but I had to make sure so I started with the mindset of discarding the others. StackStorm and Luigi were easy, the first because it is far from the market segment of my project and the second due to the complexity. Then I set to look at Node-RED and justify why I wouldn't use it instead of prefect, and, boom! it hit me in the face. Wow!

Incredible, how had I not seen this tool before? Easy to install, the debugging and simplicity mixed with the power is mind-blowing. I had an Apache Kafka consumer sending emails when I produced messages in a few clicks. What?!?

Thousands of nodes existed so I searched for Cumulocity > it is already there !!

I have to say that if, for some reason, I couldn't use Node-RED for my project I would use prefect, but I am going to use Node-RED for this one and think I will be using it a lot in the future.

Once I had watched the video series, browsed around the forum, and read most of the documentation. I implemented the lowercase development example node and did what I usually do when I find an open-source product I like:

• Translate it It is astonishing how much you can learn about an application by translating it. It is like going through every part of the application with notes. Much like reading the manual. The team at Node-RED has a nice policy where a translator needs a validator to review the translation before it can get accepted, so if any reader of this post can give me a hand there, it would be much appreciated.

• Join the community: forum and Slack.

• Implement an integration with coreBOS, obviously :-)

coreBOS Integration

The experience was, again, satisfying. The documentation, examples, community support, and overall development environment were positive. I had an integration with 11 nodes in less than a day, mostly due to my lack of comfort with node.js.

I recorded a couple of videos to present the integration.

Contenders

As I pen down these thoughts, there's a tinge of nostalgia, for it's become somewhat of a tradition to sport those coveted T-shirts proudly. Yet, I find solace in the bigger picture, where our collective efforts transcend mere attire. It's about the spirit of camaraderie and the shared purpose of making a difference in the world of technology.

This year, I decided to take a proactive approach, meticulously preparing in advance to hit the ground running when the event kicked off. The result? Lightning-fast pull requests that not only met the deadline but also left me with ample time to contribute even more throughout the month.

Here's a glimpse of the projects that received my assistance:

1. React-Admin Extensions Language Translation: I took up the task of updating the Spanish language file for the incredible React-Admin project. It may be a small win, but every contribution counts.

   https://github.com/BlackBoxVision/react-admin-extensions/pull/20

2. Tiddlywiki Language Translation: I've made a personal commitment to keep the Spanish translation of this invaluable tool up-to-date, ensuring that more people can benefit from its brilliance.

   https://github.com/Jermolene/TiddlyWiki5/pull/7761

   Speaking of Tiddlywiki, I can't help but share my enthusiasm for this cognitive extension. It's true that it has a steep learning curve, much like the tools I tend to gravitate towards, but the rewards it offers are immeasurable. It's like an extension of your mind, helping you organize and access information like never before.

3. coreBOS GenDoc Update: My home. I updated the Document Generation tool with the latest fixes and enhancements, with a special focus on supporting links within text and images, as well as support for denormalizing the Documents module.

   https://github.com/tsolucio/corebos/pull/1558

4. ADODB Issue Fix: While keeping an eye on the project that coreBOS relies on, I stumbled upon an issue that needed attention. Without hesitation, I jumped in to lend a helping hand, because that's what this initiative is all about – supporting each other.

   https://github.com/ADOdb/ADOdb/pull/1002

   https://github.com/ADOdb/ADOdb/issues/970

Yet another year of participating in this heartwarming initiative, where the joy of giving back to the open-source community knows no bounds. Here's to ten years of growth, learning, and making a difference together. Cheers to many more years of collaboration and achievement!

Happy 10th Anniversary!!

I am not into gaming, besides a brief time where I spent too much time shooting with Duke Nukem 3D, Doom, Quake, and the classics before those, I have only been addicted to Plants and Zombies. I have never sat in front of a PlayStation and barely played with the WII, so I am probably not the best person to write about the Dreamhack.

That said, I do want to share here the comparison of my past visits with this one. I have been to 3 past editions of the Dreamhack in Valencia, the last one in July 2018. Yesterday was my fourth time assisting.

What surprised me, and the reason for this post was the monotony.

I found a lot less people than in past editions. This may be due to the day I went as this year I went on the last day of the event, but we barely had to stay in the queue at the few places where we could play.

No merchandising. In the past editions, I left the event with some bags and pamphlets about all sorts of services and products. This time the only person who pushed me anything was a very friendly guy pleading with me to play his game as he only had two people in line and the basketball team looking for people to play with them. Again, this may be because it was the last day, but...

Cosplay was awesome. There were a lot of people in the role, a lot of star wars. Lara Croft was attractive and Ahsoka was very very well done! This was nice!

Lack of innovation. I spend a large part of my time reading articles about things that are happening in the (technology) world and trying software applications. I see advancements of all types in many fields, and, lately, with all the LLM hype it seems that the world is changing in gigantic steps. Yet, here, 5 years (!) later, it looked all the same; counter strike, Fortnite, GTA, brawlhalla, and League of Legends. Seriously? Still?

I mean in 2018 there were at least two stands with Virtual Reality glasses that immersed you into another world. Meta is pushing those today, yet there was only one stand and nobody was there. It was so empty that we just walked by and didn't even dare to ask.

And, augmented reality? Where is that? When the Pokemon Go boom happened in 2016 (!) I was convinced that AG was going to be predominant and ubiquitous. Fast forward 7 years (!) and it is nowhere to be seen. What happened? What is blocking that technology?

As I said at the start, I am not the right person to talk about this subject and I am, for sure, missing a big part of the picture, but it all seemed like they are living off of past work, as we do it once, get lucky and then just leave it there generating money. Not that that is bad, but it seems far from the reality I work in, where we fight hard every day for very little.

The price was too high. Not that the 15€ was expensive, everything is expensive nowadays, but for such little content and lack of entertainment, it seems like a waste of money. We arrived at 12 and had seen everything at 14h. Had lunch, walked around again, and left. So...

All in all, we had a nice time and got to spend some time together, I'm glad we went.

Self-Punishment: A Vicious Cycle

One possible explanation for engaging in insignificant activities is the tendency to punish oneself. It's not uncommon for us to feel guilty or undeserving of leisure time or relaxation, especially when we have been procrastinating for too long and have a backlog of big and urgent tasks (people waiting for us), or life interruptions keep getting in our way. Consequently, we engage in meaningless tasks as a way to subconsciously inflict punishment upon ourselves. Paradoxically, this self-punishment often results in increased feelings of dissatisfaction and unhappiness, further perpetuating the cycle. Really bad place to be.

The best advice I can give you for this situation is to avoid it completely, "Eat the frog", be conscious that sooner or later that task that is bothering you, that is making your mind escape and punish you will have to be done. Go, take a walk, take time to introspect, and identify the emotions, triggers, or thought patterns that lead you towards unproductive activities, come back and do it!

Procrastination: Avoiding the Essential

Another common culprit behind engaging in insignificant activities is procrastination. When faced with daunting or important tasks, things we really do not want to do, our minds often seek refuge in distractions. By engaging in trivial activities, we temporarily alleviate the anxiety or pressure associated with the more significant responsibilities at hand. However, this relief is short-lived, as the underlying tasks continue to linger, contributing to a sense of guilt and unproductivity and, eventually, leading us to the punishment state.

Establish clear goals and priorities to give your activities a sense of purpose. Break down larger tasks into smaller, manageable steps to minimize overwhelm and procrastination. Take control of your life and face those tasks you don't want to do, sooner or later either you will have to or the importance will fade in a way you may not like it to have happened. Life will decide for you and you may not like the decision.

Again, calm your thoughts, concentrate, and get things done!

Seeking Mental Relief and Breaks

Engaging in trivial activities may also be a sign of our minds craving relaxation or a break from the demands of our daily lives. After long work hours or periods of intense focus (and loneliness), our mental faculties can become fatigued. Engaging in mindless or insignificant activities might serve as a form of escapism, allowing our minds to unwind and recharge. However, it's important to strike a balance and ensure that these breaks do not overshadow or hinder our overall productivity.

I have heard for years that this is fixed by time management techniques. Implement effective time management techniques, such as the Pomodoro Technique, to maintain focus and productivity. By allocating specific time slots for leisure and relaxation, you can enjoy guilt-free breaks while ensuring your essential tasks are accomplished. Personally, I have never been able to make this work for me, I just keep going until I die in the boiling water. Furthermore, I still have to cross paths with someone who is actually doing one of these successfully, but if it works for you, great!

Some Guidance

Recognizing and understanding the underlying causes behind our tendencies to engage in insignificant activities is the first step towards breaking free from the cycle. In my personal case, it is all about hiding from things that overwhelm me and processing the overall loneliness.

There have been a few things that have helped me keep my sanity during these years.

• Helping people. Keeping an online activity that keeps the noise level constant during every hour of the day keeps your mind so busy that you don't have much time for anything else except eventual burnout

• Exercising has been a recent discovery that is very powerful, having that daily time away from the computer where your mind can drift into other areas is priceless. The feeling of power in your body is too :-)

• Meditating/Praying. This is something I used to do when I was much younger during a time. I am trying to get back to it now, but still have too much noise in my life to be able to appreciate the power. I see the potential, but it is a hard path to follow in today's world.

• Social Contact. This is similar to helping people but the reverse, they help you. For a brief moment of time in my life, I had a friend/coworker who was capable of snapping me out of the wasting time cycle. She would randomly catch me there and pull me out as well as expect explanations for the time I spent, which kept me focused. It was an interesting experience, a positive change. I have been working alone my whole life so I don't know much about the social dynamics of an office environment but I suppose your teammates may be able to produce a similar effect.

Engaging in insignificant activities without impact can be a frustrating and demotivating experience. While there may be various reasons behind this behavior, it's important to recognize that you have the power to break free from these cycles. That is much easier said than done!

By understanding the underlying causes and implementing strategies to regain control over your time and priorities, you can shift towards a more fulfilling and purpose-driven life. Remember, every small step towards change counts, and with determination and self-compassion, you can redirect your time and energy toward activities that truly matter to you.

Focus and take control!

When I read this sentence my mind started racing with possibilities and cases where it just didn't feel right. I want to share two ideas that were not what the author had in mind, but before doing that let me state that I understand what they are trying to say and the importance of dedicating time to those things that are important to you, be that children, family, religion, hobbies,... whatever you decide and is "conflicting" with those extra hours. As long as you are "happy", at least for now.

The first idea I want to share is that I think it is unfair in many cases. Let's imagine a gynecologist, who misses his/her daughter's dance because he is bringing new life into the world, or a firefighter who is called in for an emergency or an important meeting that will bring financial stability and a better possible future for your kids,... yes, your kids don't have those hours but you are not throwing those hours away, you are making the world a better place. Even if it isn't for such "noble" causes as the previous ones, YOU ARE doing it for a better good.

Before writing this article I had a search and found this tweet:

“20 years from now, the only people who will remember that you worked late are your kids” has both a positive and a negative connotation…

Negative: They’ll remember you missing time with them.

Positive: They’ll remember your discipline, work ethic, and energy for growth.

https://twitter.com/SahilBloom/status/1658809456375857153

That is what I am trying to share, it isn't all bad, we must read carefully and question what we see. Society makes us feel bad for doing that extra effort while at the same time, it tells us to pursue our dreams and sacrifice for a better good which only a handful of people in the world actually achieve.

Following the line of thought above I share the second idea I had. I don't believe the sentence is true, there are going to be many people who will remember those extra hours and that extra effort to help and touch other people's lives. How many families will remember that gynecologist? How many people will remember you helped them in those extra hours to get something done? That you made that extra effort to teach them something. I bet there are going to be a lot of them. Each one of those people are going to spark an energy in their brain and body and remember the moments you touched their lives, even your kids will remember the times you WERE there.

If we could capture all that energy and put it all together, it would shine like a star! Like the star that you are!

Don't let these phrases change who you are, use them to become better, think, question everything, grow and SHINE!

In the world of programming, efficient algorithms play a crucial role in determining the performance of our code. One way to analyze and predict the scalability of our code is through Big O notation. Big O provides a framework for classifying algorithms based on their runtime behavior as the input size increases. I will try to explain what this is and why I think we don't need to worry about it.

Big O Algorithm Analysis

Big O notation allows us to assess how code execution time scales with the size of the data it processes. In other words, it answers the question, "How does code slow down as data grows?" Ned Batchelder, a Python developer, beautifully described this concept in his PyCon 2018 talk titled "How Code Slows as Data Grows".

To illustrate the significance of Big O, let's consider a scenario where we have a certain amount of work that takes an hour to complete. If the workload doubles, it might be tempting to assume that it would take twice as long. However, the actual runtime depends on the nature of the work being done.

For instance, reading a short book takes an hour, and reading two short books will take approximately two hours. But if we can sort alphabetically 500 books in an hour, sorting 1,000 books will likely take longer than two hours. This is because we need to find the correct place for each book in a larger collection. On the other hand, if we are simply checking if a new book fits on the shelf, the runtime remains roughly constant, regardless of the number of books.

The Big O notation precisely captures these trends in algorithmic performance. It provides a way to evaluate how an algorithm performs regardless of the specific hardware or programming language used. Big O notation does not rely on specific units like seconds or CPU cycles but focuses on the relative growth rate of an algorithm's runtime.

Big O Orders

Big O notation encompasses various orders that classify algorithms based on their scaling behavior. Here are the commonly used orders, ranging from the least to the most significant slowdown:

1. O(1), Constant Time: Algorithms that maintain a constant runtime, regardless of input size.

2. O(log n), Logarithmic Time: Algorithms with runtime proportional to the logarithm of the input size.

3. O(n), Linear Time: Algorithms with runtime proportional to the input size.

4. O(n log n), N-Log-N Time: Algorithms with runtime proportional to the input size multiplied by the logarithm of the input size.

5. O(n^2), Polynomial Time: Algorithms with runtime proportional to the square of the input size.

6. O(2^n), Exponential Time: Algorithms with exponentially growing runtime as the input size increases.

7. O(n!), Factorial Time: Algorithms with runtime proportional to the factorial of the input size (the highest order).

The notation uses a capital O followed by parentheses containing a description of the order. For example, O(n) is read as "big oh of n" or "big oh n." It's important to note that there are more Big O orders beyond the ones mentioned here, but these are the most commonly encountered ones.

Simplifying Big O Orders

Understanding the precise mathematical meanings of terms like logarithmic or polynomial is not essential for using Big O notation effectively. Here's a simplified explanation of the different orders:

• O(1) and O(log n) algorithms are considered fast.

• O(n) and O(n log n) algorithms are reasonably efficient.

• O(n^2), O(2^n), and O(n!) algorithms are slow.

While there can be exceptions to these generalizations, they serve as helpful rules of thumb in most cases. Remember that Big O notation focuses on the worst-case scenario, describing how an algorithm behaves under unfavorable conditions.

Graphically it is also very easy to appreciate:

Some examples

O(1), Constant Time

def print_first_element(lst):
    print(lst[0])

No matter how big the list is we will always take the same amount of time to get the first element.

O(log n), Logarithmic Time

def binary_search(arr, target):
    low = 0
    high = len(arr) - 1

    while low <= high:
        mid = (low + high) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            low = mid + 1
        else:
            high = mid - 1

    return -1

This code implements the binary search algorithm, which operates on a sorted list. It repeatedly divides the search space in half until it finds the target element. The number of times we can divide any list of size n in half is log2(n) (this is simply a mathematical fact that you’d be expected to know). Thus, the while loop has a big O order of O(log n), in other words, the runtime grows logarithmically with the input size.

O(n), Linear Time

def find_max(arr):
    max_value = float('-inf')
    for num in arr:
        if num > max_value:
            max_value = num
    return max_value

This code snippet finds the maximum value in a list. It iterates through each element once, comparing it with the current maximum value. The runtime increases linearly with the input size. The more elements, the longer it takes.

O(n log n), N-Log-N Time

def binary_search(arr, target):
    low = 0
    high = len(arr) - 1
    arr.sort() # we suppose this is O(n)
    while low <= high:
        mid = (low + high) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            low = mid + 1
        else:
            high = mid - 1

    return -1

In this example, we have the same binary search we had above but the function first sorts the input list. If we suppose that the sort operation is linear with cost O(n), then the runtime of this version of the function grows in proportion to n log n.

O(n^2), Polynomial Time

def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(0, n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]

This code snippet implements the bubble sort algorithm. It compares adjacent elements and swaps them if they are in the wrong order, repeatedly iterating over the list until it becomes sorted. The runtime increases quadratically with the input size.

O(2^n), Exponential Time

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

This code calculates the Fibonacci sequence recursively. It exhibits exponential growth as the input size (n) increases because each recursive call branches into two more recursive calls, leading to a large number of redundant computations.

O(n!), Factorial Time

def permute(arr):
    if len(arr) == 0:
        return [[]]
    permutations = []
    for i in range(len(arr)):
        rest = arr[:i] + arr[i+1:]
        for p in permute(rest):
            permutations.append([arr[i]] + p)
    return permutations

This code generates all permutations of a given list. It uses a recursive approach, where it selects an element from the list and recursively generates permutations for the remaining elements. The number of permutations grows factorially with the input size.

I won't delve into further details as I believe this type of evaluation is unnecessary. Let's see why.

Considering the Size of n

While Big O notation is a powerful tool for analyzing algorithmic efficiency, it's important to remember that its significance is most apparent when dealing with a large amount of data. In real-world scenarios, the amount of data is often relatively small. In such cases, investing significant effort in designing sophisticated algorithms with lower Big O orders may not be necessary.

This is the last article in the series about optimizing and evaluating PHP applications/coreBOS performance for large datasets. I revised my knowledge of big O notation in case we could use it to evaluate some of our bottlenecks, but I understand that even with 32 million records we are still talking about small numbers for today's processing power. Our constraints are not going to be in algorithms but in business processes.

The algorithms we use and understanding big O notation will be useful to make decisions, but I am aligned with Rob Pike, the designer of the Go programming language when he expressed this idea in one of his rules of programming: "Fancy algorithms are slow when 'n' is small, and 'n' is usually small". Most developers work on everyday applications rather than massive data centers or complex computations. Profiling your code, which involves running it under a profiler, can provide more concrete insights into its performance in such contexts.

Conclusion

Understanding Big O notation and algorithm analysis empowers developers to design efficient code that can handle increasing data sizes gracefully. By classifying algorithms based on their scaling behavior, we gain insights into how different orders impact runtime. While Big O notation is most valuable when dealing with substantial data, profiling remains a practical approach for optimizing code in everyday programming tasks. By combining these techniques, developers can strike a balance between efficiency and practicality, ultimately enhancing their software's performance.

Be careful with Accidentally Quadratic code and keep security in mind!

XHGUI User Interface

At the end of our previous post, we made two profile calls to an update workflow task in coreBOS and could see them in the "Recent" tab of XHGUI.

The view shows the server name the profile was executed in, the call method used, the URL, the time, and the measurements:

• URL The URL of the request

• Time When the request was made

• wt or "Wall Time". This is the amount of time that passed during the request. It’s short for "wall clock" time, meaning the amount of time a human has to wait for the process to finish

• cpu The CPU time spent on this request

• mu Memory used for this request

• pmu The peak memory usage at any point during this request

We can sort ascending and descending by most of those columns and the next three entries in the top menu actually only do that; sort by one of the columns:

The "Custom View" tab permits us to launch some condition-based queries to get a set of raw data out of the performance measures saved in the database. I don't see the use case of this information.

"Watch functions" allow us to set function names or regular expression patterns that we want to see at the top of each run result page. It simply shows the information on those functions at the top of the call result page. Nothing really important but a nice way to visually separate some functions from the rest if necessary.

"Waterfall" deserves a section of its own, so let's leave that for later.

To get more details about a specific "run", the term used to refer to generating a response, click on the date column for the URL you are interested in. You can also click on the URL to see a list of runs and choose between them by clicking on their dates. Either way, you’ll then see a more detailed view of just this one request with a lot of information about the call.

• The top part of the screen shows the URL of the run.

• The left-hand sidebar shows full details of the call and its parameters.

• The main part of the screen shows some data about the top time-takers and memory-hoggers from all the various functions that got called during the run. There’s a detailed key below the graphs showing which bar relates to what.

• On top of the graphs, we will see information on watched functions if there are any.

• Below, we have a sorted table with all the functions called and detailed information about each one.

This table shows more detailed statistics for each of the component parts of the request. We see how many times each one was called as well as the time, CPU, and memory statistics for each function executed. Both inclusive and exclusive metrics are shown; the exclusive number gives the values for just this function whereas the inclusive values are for this function and any functions that are called from inside it. We can sort by the different values, we can filter functions and we can also click on the name of any function to get more details about that particular execution.

Another informative (and visually appealing) feature of XHGui is the Callgraph, it shows you where the time goes in a lovely visual fashion:

This shows a very nice visual hierarchy of which function calls which other function and the consumption of each function by color and size. Best of all, it’s an interactive graph, you can get more information about each function by hovering over it, click on the "view symbol" link to see the details of the function call and also click on the box itself to see a highlight of the path of the calls and a left menu with the details.

Really nice!

Comparing runs

Now we have all the information we need to optimize the code under analysis. By studying the different functions and time consumed we can set out to optimize the code and make it faster. Try to sort the functions by exclusive CPU (descending), or by memory usage, or exclusive wall time, and have a look at what lies at the top of the list. Analyze the expensive functions and try to refactor or optimize them. Also, don’t forget to check the call count; a function that is run repeatedly will deliver improvements several times over when optimized.

In order to know if we are making it faster or not we would need to be able to compare the execution before and after the optimization, and that is exactly what the "Compare Runs" functionality does for us.

Once in a run, we will find the "Compare this run" button in the upper right corner. That will permit us to select another run and retrieve a detailed output of the differences between the two executions. The summary table shows us the old and new metrics, and also the difference in both actual numbers and percentage change. The details table then gives the change in value for all the functions. You can sort by any of the columns to find the information you are looking for.

Red values are higher in 'new'. Green values are lower in 'new'.

Waterfall Display

The goal of XHGui's waterfall display is to recognize that concurrent requests can affect each other. Concurrent database requests, CPU-intensive activities, and even locks on session files can become relevant. With an Ajax-heavy application, understanding the page build is far more complex than a single load: hopefully, the waterfall can help.

This feature is important when analyzing live data from production installs where many users are executing the same code in parallel. It will segregate the information by IP and run to make it easier to understand each individual user execution.

Closing comments

In the first post of the series, I presented the PHPBench tool. This is where that tool becomes relevant because you can’t know how much you’ve improved until you start to measure your progress, which is why we must benchmark an application before proceeding with any optimizations. It’s also important to have some idea of what a realistic set of numbers should look like otherwise we may find ourselves reaching for unattainable goals. But this is also a circular relation as we need to get some idea of the bottlenecks of the application to understand which sections we need to benchmark and start optimizing.

Lastly, I must comment that we can use XHProf in production to analyze specific business procedures that are slow there. This extension is not intrusive and can be applied in a production install. That said, this should be done by informing the users during a specific time frame, not leaving it there forever. PHP will work faster if XHProf is not installed (marginally, but faster).

The goal of this series is to give you the tools you need to analyze and understand the performance of your code base and a much better insight into how it works.

Now we are ready to start applying these tools to our coreBOS 32M project!

Thanks for reading!

References

• Profiling PHP Applications with XHGui

I have to admit that I was surprised to see how little information is on the internet about this topic. Despite being a predominant language on the internet and a perfectly valid, efficient, and incredible language, PHP has no hype these days 😒.

After reading some old posts and reviewing some GitHub projects I eventually landed on the only open-source option that we have: XHProf and XHGUI

XHProf is an official PHP extension that generates tracing measurements for your PHP code. The official definition says it keeps track of call counts and inclusive metrics for arcs in the dynamic callgraph of a program. It computes exclusive metrics in the reporting/post-processing phase, such as wall (elapsed) time, CPU time, and memory usage. A functions profile can be broken down by callers or callees.

XHGUI is a web GUI for the XHProf PHP extension, using a database backend, and pretty graphs to make it easy to use and interpret.

So I started constructing our infrastructure.

XHProf

This part was very easy. XHProf is an official PHP extension so my Linux operating system just had it. I am using the exceptional ppa:ondrej/php repository which not only gives me many different PHP versions but it just has the xhprof extensions available. So all I had to do to get it working was an

apt-get install php8.0-xhprof php8.1-xhprof php8.2-xhprof

restart apache service, and there it is:

Note that there is no output_dir configured, that variable is a default value that is not needed, as we will see next.

Using XHProf

Now we have to see if it is working. Following the official PHP documentation, I create this script.

<?php
xhprof_enable(XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY);

for ($i = 0; $i <= 1000; $i++) {
    $a = $i * $i;
}

$xhprof_data = xhprof_disable();
var_dump($xhprof_data);

which gives me this output

These are the measurements that we need to feed into XHGUI but with some additional meta information. We will see how to do that after starting XHGUI, but if we wanted to save these measurements to be imported later, we could save that data into any directory we want, (which is why we don't care about the xhprof.output_dir) with some code like this

$type = 'xhproftest'; // a category for grouping
$directory = './'; // wherever we want
$file_name = $directory.uniqid().$type.'.xhprof';
$file = fopen($file_name, 'w');
if ($file) {
   fwrite($file, $xhprof_data);
   fclose($file);
}

When activating xhprof in the call above we used the predefined constants XHPROF_FLAGS_CPU and XHPROF_FLAGS_MEMORY. These are:

• XHPROF_FLAGS_NO_BUILTINS Used to skip all built-in (internal) functions.

• XHPROF_FLAGS_CPU Used to add CPU profiling information to the output.

• XHPROF_FLAGS_MEMORY Used to add memory profiling information to the output.

So you can set any of those as per your requirements.

Now let's see how to visualize that data.

XHGUI

The instructions on the project GitHub page are clear and simple but I decided to try the docker install as it is much easier and cleaner to roll back in case I wasn't convinced. So I cloned the repository and read the docker-compose.yml file to make sure it wasn't going to do anything strange and started it.

The output of the docker-compose command was all correct and I could access the web user interface at http://localhost:8142

So, that easy, we are ready to start sending the profiling measurements there.

Using PHP Profiler

PHP Profile is a PHP profiling library to submit profilings to XHGui. This library encapsulates the logic described above and gives us some high-level functions to work with profiling. This library does all the heavy lifting for us so we don't have to make calls to the database directly nor have to call the xhprof enable and disable. It gives a nice abstraction to do profiling instead of having to do low-level tasks.

I read the documentation and installed the library with

composer require perftools/php-profiler

Next, I read the configuration file and finally created this code which is equivalent to the first profiling code we had above.

<?php
require_once 'vendor/autoload.php';
use Xhgui\Profiler\Profiler;
use Xhgui\Profiler\ProfilingFlags;

$config = array(
    // This allows to configure, what profiling data to capture
    'profiler.flags' => array(
        ProfilingFlags::CPU,
        ProfilingFlags::MEMORY,
        ProfilingFlags::NO_BUILTINS,
        ProfilingFlags::NO_SPANS,
    ),

    // Saver to use.
    'save.handler' => Profiler::SAVER_UPLOAD,

    'save.handler.upload' => array(
        'url' => 'http://localhost:8142/run/import',
        // The timeout option is in seconds and defaults to 3 if unspecified.
        'timeout' => 3,
        // the token must match 'upload.token' config in XHGui
        'token' => 'token',
    ),
);

$profiler = new Profiler($config);
$profiler->enable([]);

for ($i = 0; $i <= 1000; $i++) {
    $a = $i * $i;
}

$profiler_data = $profiler->disable();
$profiler->save($profiler_data);
var_dump($profiler_data);

Let's go over that.

• I load the library and namespaces

• I create a configuration array variable.

  • I set the equivalent flags that we had in our first script

  • I define where the library should save the measurements (SAVER_UPLOAD). This requires some attention. XHGUI, which we started with docker, supports the endpoint run/import so we can call that from the PHP Profiler library, but both applications MUST share a secret token. I had to stop the docker-compose and add the XHGUI_UPLOAD_TOKEN environment variable with the value token

• I initialize the profiler library with the configuration

• I start and end profiling

• I save the data in the XHGUI import endpoint which ends up in the MongoDB database

• I dump the data and see that it is richer than the first test we did

I run that twice and reload the XHGUI web interface:

This is the change I made in the docker-compose file

diff --git a/docker-compose.yml b/docker-compose.yml
index e5efbd6..3132828 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -11,6 +11,7 @@ services:
     environment:
       - XHGUI_MONGO_HOSTNAME=mongo
       - XHGUI_MONGO_DATABASE=xhprof
+      - XHGUI_UPLOAD_TOKEN=token
     ports:
       - "8142:80"

Convinced!

coreBOS

With this knowledge, adding profiling to coreBOS is simple.

• I use composer to add the PHP Profile library.

• I create a script to load the configuration and the object in build/ProfilerConfig.php which looks like this

<?php
require_once 'vendor/autoload.php';
use Xhgui\Profiler\Profiler;
use Xhgui\Profiler\ProfilingFlags;

$ProfileConfig = array(
    // This allows to configure, what profiling data to capture
    'profiler.flags' => array(
        ProfilingFlags::CPU,
        ProfilingFlags::MEMORY,
        ProfilingFlags::NO_BUILTINS,
        ProfilingFlags::NO_SPANS,
    ),

    // Saver to use.
    'save.handler' => \Xhgui\Profiler\Profiler::SAVER_UPLOAD,

    'save.handler.upload' => array(
        'url' => 'http://localhost:8142/run/import',
        // The timeout option is in seconds and defaults to 3 if unspecified.
        'timeout' => 3,
        // the token must match 'upload.token' config in XHGui
        'token' => 'token',
    ),
);
$profiler = new Profiler($ProfileConfig);

Obviously, you have to set the configuration to your environment.

Now we can include this script and profile any code we need with just three lines of code. I tested that with the workflow Update task. I added the three lines to the modules/com_vtiger_workflow/tasks/VTUpdateFieldsTask.inc script; two at the start of the doTask() method, to include the script above and start profiling, and the last line at the end of the method to stop profiling and save.

diff --git a/modules/com_vtiger_workflow/tasks/VTUpdateFieldsTask.inc b/modules/com_vtiger_workflow/tasks/VTUpdateFieldsTask.inc
index ed6a16e54..bdb30a59d 100644
--- a/modules/com_vtiger_workflow/tasks/VTUpdateFieldsTask.inc
+++ b/modules/com_vtiger_workflow/tasks/VTUpdateFieldsTask.inc
@@ -29,6 +29,8 @@ class VTUpdateFieldsTask extends VTTask {

        public function doTask(&$entity) {
                global $adb, $current_user, $logbg, $from_wf, $currentModule;
+               include 'build/ProfileConfig.php';
+               $profiler->enable([]);
                $logbg->debug('> UpdateFieldsTask');
                $from_wf = true;
                $util = new VTWorkflowUtils();
@@ -183,6 +185,7 @@ class VTUpdateFieldsTask extends VTTask {
                $util->revertUser();
                $from_wf = false;
                $logbg->debug('< UpdateFieldsTask');
+               $profiler->save($profiler->disable());
        }
 }
 ?>

So we have to include the profiling configuration and start profiling before the code we want to measure and stop profiling and save after it:

// ...
include 'build/ProfileConfig.php';
$profiler->enable([]);
// code we want to measure
$profiler->save($profiler->disable());
// ...

Now we are ready to start measuring the performance of coreBOS

Summary

• I understand better now why there isn't much information out there, it is really easy to get this working and there aren't any other options except the paid ones

• we need three tools working together to do profiling: XHProf, PHP Profile, and XHGUI

• getting them to work together is relatively easy

• I will do another post explaining how to use the XHGUI information we are saving

Thanks for reading.

References

• coreBOS

• XHProf

• XHGUI

• PHP Profile

Studying the different options, I found PHPBench, a benchmark runner for PHP analogous to PHPUnit but for performance, and decided to give it a try.

The documentation is spot-on, clear, direct, and just what you need. Getting the tool configured and working is a breeze.

The set of options and functionality is amazing and simple. It just works.

After reading the documentation I decided to follow the same approach coreBOS already has with the unit test project, because it is a separate project that you can apply on any coreBOS install that needs to do the unit tests and I understand that we will be doing performance benchmarking also only on a handful of installs.

So I created a separate repository and downloaded the PHPBench phar file into it.

curl -Lo phpbench.phar https://github.com/phpbench/phpbench/releases/latest/download/phpbench.phar

I validated it as per the instructions and pushed it to the repository.

Now we can start benchmarking some code.

The first challenge is to load the coreBOS infrastructure so that we can access all the awesome functionality. Turns out that the test project also had that problem and solved it by creating a generic "load everything" script. So I just copied that one from there and included it at the top of the benchmark script.

I followed the Quick Start instructions and create this script that executes two functions, 5 times in batches of 1000 runs.

<?php
include_once 'build/evBench/loadcorebos.php';

class CommonUtilsBench {

    /**
    * @Revs(1000)
    * @Iterations(5)
    */
    public function benchgetCurrencyName() {
        getCurrencyName(1, true);
    }

    /**
    * @Revs(1000)
    * @Iterations(5)
    */
    public function benchgpopup_from_html() {
        popup_from_html('$string', true);
    }
}

I ran the benchmarks with this command

build/evBench/phpbench.phar run build/evBench/include/utils/CommonUtils.php --report=aggregate

and received this output

PHPBench (1.2.10) running benchmarks... #standwithukraine
with PHP version 8.2.7, xdebug ✔, opcache ❌

\CommonUtilsBench

    benchgetCurrencyName....................I4 - Mo1.801μs (±9.46%)
    benchgpopup_from_html...................I4 - Mo1.729μs (±1.13%)

Subjects: 2, Assertions: 0, Failures: 0, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.985mb | 1.801μs | ±9.46% |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.729μs | ±1.13% |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+

That simple! Really, really nice!

Assertions

Another awesome feature that I found was Assertions. You can annotate your PHPBench scripts with a powerful expression language to validate the timing of your functions and then add these benchmarks to your CI/CD process to detect variations in the execution time of the critical functions of your code base.

I tried that by adding this validation to the two functions in the script above.

* @Assert("mode(variant.time.avg) < 200 ms")

and received this output

    benchgetCurrencyName....................I4 ✔ Mo1.826μs (±2.06%)
    benchgpopup_from_html...................I4 ✔ Mo1.757μs (±8.47%)

Subjects: 2, Assertions: 2, Failures: 0, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.985mb | 1.826μs | ±2.06% |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.757μs | ±8.47% |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+

Notice the new (green) line above the table

Subjects: 2, Assertions: 2, Failures: 0, Errors: 0

Next, I tried to force an error with this assertion

* @Assert("mode(variant.time.avg) < 1 microsecond")

and got this response

    benchgetCurrencyName....................I4 ✘ Mo1.791μs (±2.95%)
    benchgpopup_from_html...................I4 ✘ Mo1.726μs (±24.63%)

2 variants failed:

  ✘ \CommonUtilsBench::benchgetCurrencyName # 

    1) mode(variant[time][avg]) < 1 microsecond
       = 1.791373776908 < 1 microsecond
       = false

  ✘ \CommonUtilsBench::benchgpopup_from_html # 

    1) mode(variant[time][avg]) < 1 microsecond
       = 1.726025440313 < 1 microsecond
       = false

Subjects: 2, Assertions: 2, Failures: 2, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+---------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev  |
+------------------+-----------------------+-----+------+-----+----------+---------+---------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.986mb | 1.791μs | ±2.95%  |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.726μs | ±24.63% |
+------------------+-----------------------+-----+------+-----+----------+---------+---------+

Simple and effective!

This is what the final code looked like

    /**
    * @Revs(1000)
    * @Iterations(5)
    * @Assert("mode(variant.time.avg) < 20 ms")
    */
    public function benchgetCurrencyName() {
        getCurrencyName(1, true);
    }

Hard Part

Now we have the infrastructure to measure the performance of functions and processes inside the coreBOS application, but we have to continue asking ourselves, "Which are the methods and processes that we should measure?", "Where are the bottlenecks of our code?", "What code do we have to keep under control?"

Stay tuned as I continue to create the infrastructure to answer those questions.

Thanks for reading.

References

• coreBOS

• PHPBench

• PHPBench Documentation

• coreBOS Benchmark repository