[{"categories":null,"content":"Photo by Bruno Kelzer on Unsplash A while back, I was working on a project for a client. Day to day, I\u0026rsquo;m a backend engineer, but in this project the scope expanded. I had to dip into the server side as well. The client wanted to migrate their Laravel application from AWS Elastic Beanstalk to AWS Lightsail, aiming to cut infrastructure costs and gain more control over the server.\nAs someone who enjoys learning, this was actually an exciting moment. I\u0026rsquo;d been comfortable in the backend zone, and now I had the chance to get my hands on infrastructure and DevOps work directly.\nBeanstalk had been incredibly convenient, the entire deployment process was automated. Push code, done. But once we moved to Lightsail, everything changed. Suddenly I had to set up the server from scratch, configure Docker, and the most frustrating part: a completely manual deployment process.\nThe Problem: Manual Deployment That Took Forever In the initial Lightsail setup, every time I wanted to deploy changes to production, here\u0026rsquo;s the ritual I had to go through:\nPush commits to GitHub Create a release tag (git tag -a v1.2.3 -m \u0026quot;release note\u0026quot; \u0026amp;\u0026amp; git push origin v1.2.3) SSH into the Lightsail server git fetch --tags and git checkout the new tag Run docker compose up -d --build If there were database changes, hop into the container with docker exec then manually run php artisan migrate --force All of these steps took about 2-3 minutes. And when you\u0026rsquo;re rushing because there\u0026rsquo;s a critical bug in servers that need to be fixed quickly, every second felt painfully slow.\nBut the real issue wasn\u0026rsquo;t the duration. It was the sheer number of manual steps, each one a potential point of human error. Once, I forgot to run the migration after deploying, and immediately got errors because a new database column didn\u0026rsquo;t exist yet. Pretty embarrassing, all because I was in a hurry and there were too many steps.\nSo the question popped up: why not just automate all of this? Back when I was working at a startup, the infra team had already set up a pipeline like this. How about I figure it out and build it myself?\nThe Solution: GitHub Actions Workflow Since the code was already on GitHub and the server was using Docker, the answer was clear: GitHub Actions. With workflow_dispatch, we can create a workflow triggered manually from the Actions tab — complete with input parameters.\nYou might be wondering, why not trigger it automatically on every commit or merge to main? Wouldn\u0026rsquo;t that be more convenient?\nHere\u0026rsquo;s the reasoning behind the manual trigger:\nNo surprise changes in production. Every deployment must be intentional, not a side effect of a merge The whole team stays aware because before deploying, the person responsible has to go to the Actions tab and run it consciously Versioning is mandatory — with the tag input, we\u0026rsquo;re forced to create a version for every deployment. If something goes wrong down the line, we can roll back to a previous tag Alright, so the concept is simple:\nPush commits and create tags as usual Open the Actions tab in the GitHub repository Select the Deploy Production workflow Enter the tag you want to deploy Check the box if you want to run database migrations Click Run workflow, and\u0026hellip; just wait No more SSH, no more docker exec, no more worrying about forgetting migrations. Deployments that used to take 2-3 minutes now finish in an average of 40 seconds (most of that is waiting for Docker to build).\nLet\u0026rsquo;s take a look at how this workflow is built.\nBuilding the Workflow The workflow lives at .github/workflows/deploy.yml. Here\u0026rsquo;s the basic structure:\nname: Deploy Production on: workflow_dispatch: inputs: tag: description: \u0026#34;Git tag to deploy (example: v1.2.3)\u0026#34; required: true type: string run_migration: description: \u0026#34;Run database migration?\u0026#34; required: false type: boolean default: false With the workflow_dispatch configuration above, GitHub will display a simple form every time we want to run the workflow. There are two inputs:\ntag, i.e. the Git tag to deploy, required run_migration, i.e. a checkbox to run database migrations, optional Pretty straightforward. Now let\u0026rsquo;s dive into the jobs section.\nSSH into the Server The first step is setting up an SSH connection to the Lightsail server using an SSH key stored as a GitHub Secret:\njobs: deploy: runs-on: ubuntu-latest steps: - name: Setup SSH run: | mkdir -p ~/.ssh echo \u0026#34;${{ secrets.SSH_PRIVATE_KEY }}\u0026#34; \u0026gt; ~/.ssh/id_ed25519 chmod 600 ~/.ssh/id_ed25519 ssh-keyscan -H ${{ secrets.SSH_HOST }} \u0026gt;\u0026gt; ~/.ssh/known_hosts Here we\u0026rsquo;re saving the private key to id_ed25519 with permission 600 (readable only by the owner). Then we add the server\u0026rsquo;s host key to known_hosts so the SSH connection isn\u0026rsquo;t rejected.\nKeep in mind, all sensitive information like SSH_PRIVATE_KEY, SSH_HOST, SSH_USER, and APP_PATH should be stored as GitHub Secrets. Never hardcoded in the workflow file.\nCheckout Tag and Deploy Once the SSH connection is ready, we send commands to the server to fetch tags, checkout, and build with Docker:\n- name: Deploy to Server run: | ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }} \u0026#34;export GITHUB_PAT=\u0026#39;${{ secrets.CREDENTIAL_GITHUB_PAT }}\u0026#39; \u0026amp;\u0026amp; bash -s\u0026#34; \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; set -e set +o history cd ${{ secrets.APP_PATH }} echo \u0026#34;Temporarily set Git remote with token\u0026#34; ORIGINAL_URL=$(git remote get-url origin) REPO_PATH=${ORIGINAL_URL#https://github.com/} git config --local credential.helper \u0026#39;!f() { echo \u0026#34;username=x-access-token\u0026#34;; echo \u0026#34;password=${GITHUB_PAT}\u0026#34;; }; f\u0026#39; echo \u0026#34;Fetch tags\u0026#34; git fetch --tags echo \u0026#34;Checkout tag: ${{ github.event.inputs.tag }}\u0026#34; git checkout -f ${{ github.event.inputs.tag }} echo \u0026#34;Remove credential helper\u0026#34; git config --local --unset credential.helper echo \u0026#34;Deploy with Docker Compose\u0026#34; export USER_ID=$(id -u) export GROUP_ID=$(id -g) docker compose up -d --build set -o history echo \u0026#34;Deployment completed successfully!\u0026#34; EOF There are a few interesting things here. Let\u0026rsquo;s break them down.\nFirst, we\u0026rsquo;re using a heredoc (\u0026lt;\u0026lt; 'EOF') to send a block of shell commands to the server over SSH. This is much cleaner than writing everything inline in one long line.\nSecond, we\u0026rsquo;re using a Git credential helper instead of storing the token in the remote URL. This is more secure because the credential helper is removed right after git checkout completes. The token only lives for the duration of that command.\nThird, notice set +o history at the top. This disables bash history temporarily so the GitHub token doesn\u0026rsquo;t get recorded in the server\u0026rsquo;s shell history. A small security detail that\u0026rsquo;s often overlooked.\nFourth, export USER_ID and GROUP_ID are needed so the Docker container runs with the same user ID as the host, keeping file permissions consistent.\nRunning Migrations The final piece is the option to run database migrations:\n# Run migration if requested if [ \u0026#34;${{ github.event.inputs.run_migration }}\u0026#34; == \u0026#34;true\u0026#34; ]; then echo \u0026#34;Running database migration...\u0026#34; docker exec container_app php artisan migrate --force echo \u0026#34;Migration completed!\u0026#34; else echo \u0026#34;Skipping migration\u0026#34; fi Pretty straightforward. Check if the run_migration input is true. If yes, run php artisan migrate --force inside the container_app container. The --force flag is needed because Artisan usually prompts for confirmation in production environments.\nThis part used to be the most annoying. Often, right after docker compose up -d finished, I had to quickly docker exec to run migrations before any requests hit the new tables. Now it\u0026rsquo;s just ticking a box and we\u0026rsquo;re done.\nThe Final Result With the workflow ready, deploying now is as simple as:\nPush commits and create a tag (git tag v1.2.3 \u0026amp;\u0026amp; git push origin v1.2.3) Open the repository on GitHub, Actions tab → select Deploy Production → click Run workflow Enter the tag you want to deploy Check Run database migration? if there are database changes Click Run workflow and wait for the success notification Here\u0026rsquo;s the before-and-after comparison:\nStep Before (Manual) After (GitHub Actions) Checkout code on server SSH → git fetch → git checkout Automated via workflow Build \u0026amp; deploy docker compose up -d --build Automated via workflow Database migration SSH → docker exec → php artisan migrate Check a checkbox Total time 2–3 minutes ~40 seconds (input params) Human error risk High (missed steps) Low (standardized) From 2-3 minutes down to ~40 seconds — that\u0026rsquo;s roughly an 80% reduction in deployment time. But more important than the time savings is this: deployments are now consistent and free from human error.\nConclusion Migrating from Beanstalk to Lightsail definitely pushed me out of my backend engineer comfort zone and into infrastructure territory I\u0026rsquo;d previously left to automated services, deployment included. But that\u0026rsquo;s exactly where the learning happens.\nWith GitHub Actions workflow_dispatch, we can build a simple CI/CD pipeline that\u0026rsquo;s just as convenient as managed services like Beanstalk. All you really need is:\nA repository on GitHub A server accessible via SSH An application containerized with Docker A single workflow YAML file Sure, this pipeline may not be as sophisticated as Jenkins or GitLab CI with automatic rollback and approval gates. But for small teams or personal projects, it\u0026rsquo;s more than enough.\nIf you have any additions or corrections to the discussion above, let\u0026rsquo;s talk in the comments. Hope this helps 👋.\n","date":"2026-06-03T10:00:00+07:00","description":"How I cut deployment time by 80% by replacing a slow manual process with GitHub Actions and Docker.","permalink":"https://rizkynotes.com/en/posts/cutting-deployment-time-by-80-with-github-actions-and-docker/","tags":["github-actions","docker","ci-cd","devops","laravel"],"title":"Cutting Deployment Time by 80% with GitHub Actions and Docker"},{"categories":null,"content":"Photo by Javier Allegue Barros on Unsplash Api gateways are an important component in modern software architecture, especially in systems that implement microservices. Api Gateway acts as the main gateway to receive all incoming API requests. With this role, API Gateway simplifies API management and improves overall system performance and security.\nUsing API Gateway helps build systems that are scalable and easy to maintain. As system complexity increases, API Gateway plays a critical role in managing effective integration and communication between various backend services and clients.\nIn this article, we will learn basic concepts and benefits from using API Gateway. The hope is that this article can provide new understanding or strengthen the insights you already have.\nWhat is API Gateway ? An API Gateway is an API management tool that sits between a client and a collection of backend services. It acts as an intermediary, routing client requests to the appropriate backend service required to fulfill them and then returning the corresponding response Additionally, an API Gateway provides several important features such as load balancing, circuit breaking, logging, authentication, and caching, making it a crucial component in modern API architecture.\nIn today\u0026rsquo;s fast-paced world of digital development, efficiency is everything, which is why API Gateways have become essential in modern software projects. Acting as a central point of control for API across microservices architecture handling thousands of concurrent API calls efficiently.\nHow does API Gateway work ? Api Gateway work with receive request from internal and external, called \u0026ldquo;API Calls\u0026rdquo;. API Gateway as software layer that manage traffic for routes them ti appropriate API and then delver the responses to the particular user or devic thath made the request.\nThis the explain API Gateway Workflow from image above:\nA client (browser, mobile app, or another service) sends an API request. The API Gateway processes the request with routes the request to the correct backend service. The backend service processes the request and sends a response. The API Gateway modifies or enhances the response. Finally, the response is returned to the client. Benefits of API Gateway Using an API Gateway can bring several benefits such as enhance performance, improve security, easier maintenance of multiple backend services. Here are some key benefits of using an API Gateway:\n1. Enhanced security An API Gateway enforce authentication and authorization policies, ensuring that only authorized users can access backend services. Additionally, it can help mitigate security threats through:\nRate limiting to prevent API abuse and DDoS attacks. IP whitelisting and blacklisting to restrict access. TLS encryption to secure communication between clients and services. 2. Better monitoring and visibility An API gateway can collect metrics and other data about the requests and responses, providing valuable insights into the performance and behavior of the system. This can help to identify and diagnose problems, and improve the overall reliability and resilience of the system.\n3. Data Format Transformation An API Gateway can convert request and response or data data format (e.g., JSON to XML). Making it easier to manage data formats when needed.\n4. API Versioning and Backward Compatibility An API Gateway can manage multiple versions of an API, allowing developers to introduce new features or make changes without breaking existing clients. This enables a smoother transition for clients and reduces the risk of service disruptions.\n5. Enhanced Error Handling An API Gateway can provide a consistent way to handle errors and generate error responses, improving the user experience and making it easier to diagnose and fix issues.\nDrawbacks of Impelementation API Gateway In addition to having many benefits that we get when we implement API Gateway, on the other hand we also have disadvantages in using it, here are some points:\n1. Single Point of Failure Since the API Gateway acts as a central access point for all API requests, it can become a single point of failure if not properly managed. If the gateway experiences downtime, it could disrupt the entire API ecosystem.\n2. Additional Complexity Integrating an API Gateway introduces an additional layer of complexity to your architecture. It requires careful configuration, monitoring, and maintenance. If not managed efficiently, it can slow down development and increase operational overhead.\n3. Increased Latency An API Gateway processes incoming requests before forwarding them to the appropriate services, which can introduce latency. This delay may impact overall system performance, especially if multiple processing steps (such as authentication, logging, and rate limiting) are involved.\n4. Cost Operating an API Gateway, particularly in high-traffic environments, can add significant costs to your infrastructure. Expenses may include hosting, licensing fees, or managed API Gateway services from cloud providers.\nSimple Example of an API Gateway To implement an API Gateway, you can use a popular API Gateway such as Nginx, Kong, KrakenD, and many more popular choices. However, our goal for the topic is to understand how API Gateway works. So, we can understand by simply building a simple API Gateway using golang. Let\u0026rsquo;s begin.\nFor the complete code you can see following repository: Click me\nThe structure Folder We can create a simply the API Gateway following the structure folder:\n├── Makefile ├── api_gateway │ ├── go.mod │ ├── go.sum │ └── main.go ├── service_cart │ ├── go.mod │ ├── go.sum │ └── main.go ├── service_order │ ├── go.mod │ ├── go.sum │ └── main.go └── service_product ├── go.mod ├── go.sum └── main.go The rate limitter function code First, we can create a file with the name main.go in the api_gateway folder which contains:\npackage main import ( \u0026#34;time\u0026#34; \u0026#34;github.com/gofiber/fiber/v2\u0026#34; \u0026#34;github.com/gofiber/fiber/v2/middleware/limiter\u0026#34; ) func rateLimiter() fiber.Handler { return limiter.New(limiter.Config{ Expiration: time.Second * 60, Max: 5, }) } The rateLimiter() function is used to limit the number of requests per client. Where this rate limiter is useful for Preventing abuse \u0026amp; DDoS, Controlling API traffic, Protecting backend services from overload.\nWith the configuration Expiration: time.Second, Max: 5, each client can only make 5 requests per second. If this limit is reached, the client will receive error HTTP 429 Too Many Requests. After 1 second, the limit will be reset and the client can send requests again.\nThe proxy function code Second, we also create a function with the name reverseProxy:\npackage main import ( // ... other import libraries \u0026#34;log\u0026#34; \u0026#34;github.com/gofiber/fiber/v2/middleware/proxy\u0026#34; ) // ... rate limitter function func reverseProxy(target string) fiber.Handler { return func(c *fiber.Ctx) error { url := target + c.OriginalURL() return proxy.Do(c, url) } } The proxy(target string) function is used to forward requests (proxy requests) from clients to other backend servers specified in the target. This function returns a middleware handler (fiber.Handler) that can be used on Fiber.\nWe also see c.OriginalURL() returns the path and query parameters of the original request sent by the client. For example, if the client accesses http://localhost:3000/api/orders?id=10, then:\n/api/orders?id=10 And last we return proxy.Do(c, url) is used to forward requests to the destination url. This url is a combination of target and c.OriginalURL(), so all incoming requests will be directed to another server with the same path. In this way, requests that enter the API Gateway will be automatically directed to the appropriate backend service without the client knowing.\nThen main function code And latest file we create main function, is a entrypoint of this application. In this case, it acts as an API Gateway. The code is like this:\npackage main import ( // ... other import libraries \u0026#34;github.com/gofiber/fiber/v2/middleware/logger\u0026#34; ) func main() { app := fiber.New() app.Use(logger.New()) // Implement rate limitter app.Use(rateLimiter()) // Implement proxy // Redirect request to service product app.Use(\u0026#34;/product\u0026#34;, reverseProxy(\u0026#34;http://localhost:5001\u0026#34;)) // Redirect request to service order app.Use(\u0026#34;/order\u0026#34;, reverseProxy(\u0026#34;http://localhost:5002\u0026#34;)) // Redirect request to service cart app.Use(\u0026#34;/cart\u0026#34;, reverseProxy(\u0026#34;http://localhost:5003\u0026#34;)) port := \u0026#34;:3000\u0026#34; log.Printf(\u0026#34;Starting API Gateway in port %s\u0026#34;, port) log.Fatal(app.Listen(port)) } We can focus on the code part:\n// Implement rate limitter app.Use(rateLimiter()) This code applies rate limiting to the entire application with Fiber\u0026rsquo;s limiter middleware. This middleware is attached with app.Use(rateLimiter()), which means it will apply to all routes. The function rateLimiter() returns the rate limiter middleware we created earlier with limiter.New().\nAnd also for some of these code:\n// Implement proxy // Redirect request to service product app.Use(\u0026#34;/product\u0026#34;, reverseProxy(\u0026#34;http://localhost:5001\u0026#34;)) // Redirect request to service order app.Use(\u0026#34;/order\u0026#34;, reverseProxy(\u0026#34;http://localhost:5002\u0026#34;)) // Redirect request to service cart app.Use(\u0026#34;/cart\u0026#34;, reverseProxy(\u0026#34;http://localhost:5003\u0026#34;)) This code uses the reverseProxy function we created earlier, which will forward the request to the appropriate service.\nAs we know in the The structure Folder section, we also have several service folders that will be directed according to incoming requests from clients.\nFor the complete code you can see following repository: Click me\nLet\u0026rsquo;s run it To run it there are several ways, if Makefile is installed on your computer you only need to run the command below in the terminal on root folder project:\nmake run If not you can run it one by one by going into all folders and starting to run with the command:\ncd service_product \u0026amp;\u0026amp; go run main.go cd service_order \u0026amp;\u0026amp; go run main.go cd service_cart \u0026amp;\u0026amp; go run main.go cd api_gateway \u0026amp;\u0026amp; go run main.go If successful, you can see display like this:\nFrom the image above you can see several services running on their respective ports.\nService API Gateway running on port 3000 Service product running on port 5001 Service order running on port 5002 Service cart running on port 5003 Here is the advantage of using API Gateway, imagine you have more than 3 services and on the client we have several features that must be run simultaneously. Do we have to define the url with each specific port? of course it is very troublesome. And later one of the services must change its url address, of course it is even more troublesome to have to refactor here and there on the client side.\nBut with API Gateway you only need to know the url on the API Gateway and define the appropriate url, let\u0026rsquo;s try to access several services via the API Gateway url only. Run this command:\ncurl http://localhost:3000/product/1 curl http://localhost:3000/order curl http://localhost:3000/cart And the result is you can access multiple services with just one URL.\nAnd in the log you can see that the api gateway actually forwards the request to the service that corresponds to the address being called.\nAnd I give a challenging task, try to make more requests than the maximum rate limiter that we have set in the rateLimiter function until you get the error HTTP 429 Too Many Requests.\nConclusion API Gateway acts as the primary gateway to manage request traffic in a microservice system. With rate limiting, APIs are more secure from abuse such as DDoS. Reverse proxies allow requests to be directed to the appropriate backend service without changing the client side, increasing flexibility and ease of management. In addition, API Gateway also supports caching, logging, monitoring, and other security features such as authentication \u0026amp; authorization. With proper implementation, API Gateway can improve the security, performance, and scalability of the overall system.\nHowever, implementing API Gateway also has some drawbacks that we must be aware of such as a single point of failure, additional complexity, increased latency, and higher operational costs. Despite these drawbacks, a well-implemented API Gateway remains essential for scalable and secure microservice architectures.\nHopefully this article can help to improve understanding or recall what is already known. If you have additions or corrections to the discussion above, let\u0026rsquo;s discuss it in the comments column. Hopefully it helps 👋.\nReading References What is an API Gateway? What does an API gateway do? Advantages and disadvantages of using API gateway Deciphering the Decision: Should You Use an API Gateway or Not? ","date":"2025-03-27T09:57:41+07:00","description":"The basic concepts and benefits of an API Gateway in modern microservices architecture.","permalink":"https://rizkynotes.com/en/posts/basic-concept-and-benefit-of-api-gateway/","tags":["tips","backend engineering","System Design"],"title":"Basic Concept \u0026 Benefit of API Gateway"},{"categories":null,"content":"Photo by Christine Tutunjian on Unsplash Connection Pooling is a mechanism that creates and manages a pool of database connections that can be used by applications. This concept is important in managing connections to the database with the aim of optimizing resource use and improving the performance of applications that frequently interact with the database.\nInstead of creating a new connection every time it is needed (which is expensive in terms of time and resources), connection pools allow applications to borrow/use existing connections and return them to the pool when they are finished using them. That\u0026rsquo;s why it\u0026rsquo;s called connection pool.\nWhy connection pool is important ? The important question is why is connection polling important? why not just make 1 connection to be used alternately. Let\u0026rsquo;s discuss 1 by 1 why connection pooling is important.\nBetter performance Opening a new connection to the database can take time because it goes through several processes such as authentication, network settings and so on. With a connection pool, applications can avoid this process because connections are already available if needed.\nResource savings Without a pool, if the application gets 100 requests, the application will create 100 new connections. This can burden the database server which is always busy carrying out the initialization process. The connection pool can limit the number of connections used simultaneously, this keeps the database server load stable.\nScalability Connection pooling helps applications manage load well when the application scales up. With good connection pooling settings, we can handle a large number of requests coming at the same time and ensure the application remains responsive.\nLoad configuration The connection pool can be configured with a maximum number of connections according to needs. So it can prevent applications from flooding connections to the database.\nHow to Works ? Now that we understand why connection pooling is important to use, we will now discuss how connection polling works.\nPool Initialization The first time an application is run, it makes a certain number of connections to the database (for example, 10 connections). This number is called Pool size. This connection will be idling waiting to be used.\nConnection usage When the application needs a connection to the database it no longer creates a new connection, instead it requests a connection from the pool. If a connection is available in the pool, it will be assigned to the application. If no connections are available, the application may have to wait until a connection is released and returns to the pool after being used by another process.\nRestore connection Once an application has finished using a connection, it will not close the connection. Instead, the connection is returned to the pool to be used again by the next request.\nThe common main parameter in connection pool The following are some of the main parameters that are commonly used when creating a connection pool. I will try to explain in detail followed by analogies that can help us understand each context better.\nPool Size We can determine the number of connections provided in Pool Size. If all connections are used then when there is a request to use the connection it will queue up waiting for an available connection or it will fail if the queue is full.\nFor example, if we set the max pool size to 10, only 10 active connections can be used by the application. If an 11th request comes, this request will wait for a connection from the pool that is already in use. The analogy is like this:\nImagine there is a parking lot, if all the slots are full, a new car that wants to enter (connection request) has to wait until another car comes out (idle connection). If we want to accommodate more cars, we have to build more parking slots (increase MaxPoolSize) so that we can accommodate more cars. But there are increased maintenance costs (server resources). Minimum Connections Minimum connection is the minimum number of connections that will be created the first time the application is run. This is a connection that will always be on standby waiting to be used, even when there are no requests. For example, if we set minimum pool size 5, the pool will ensure there are always 5 idle connections. Even if there are no active requests, the pool will still make sure these 5 connections are open to reduce the connection opening time when a request comes in.\nThis is useful for anticipating erratic loads, for example suddenly our application gets a lot of requests and connections are available ready to use without having to open a new connection. The analogy is like this:\nImagine that a restaurant has 10 chairs that are deliberately not arranged in the dining room, but are stored in the warehouse. But when there are a lot of visitors, we need extra chairs, we don\u0026rsquo;t need long to buy them at the shop. But just need to go to the warehouse and pick up the 10 chairs needed for use. Idle Connection Idle connections are the number of connections that are not being used but remain open in the pool, which can also be called \u0026ldquo;idle connections\u0026rdquo;. This parameter is usually set by:\nMaximum Idle Time: How long an idle connection is maintained before being closed. Maximum Idle Connections: The number of idle connections allowed to remain in the pool. For example, if we set MaxIdleConnections = 5, the pool will maintain a maximum of 5 idle connections. If there are more idle connections it will be closed. If there are no new requests within a certain time (for example, 10 minutes), these idle connections may be removed to save resources. The analogy is like this:\nIn the supermarket there are 5 cashiers, 3 cashiers are serving buyers (active connection) and 2 cashiers are on standby in their positions without a queue of buyers (idle connection) and are ready whenever a buyer comes. Supermarkets have a rule, \u0026ldquo;If the cashier is on standby for 30 minutes without a customer, then they can go home.\u0026rdquo; These 30 minutes are idle time. Timeout We can set the maximum time the application waits for a connection from the pool before it fails (error). This is done to prevent applications from waiting too long for available connections from the pool, if the pool is busy or full. This is useful to prevent the application from waiting too long for a connection which will cause the application to \u0026ldquo;freeze\u0026rdquo; where it is better to give an error to the user, so that it can provide information to the user to try again rather than resulting in a freeze request.\nFor example, if we set a timeout of 5 seconds, if a request does not get a connection, after waiting for 5 seconds it will receive an error. The point to remember is that a timeout that is too short can cause the request to fail too quickly and a timeout that is too long means the request will hang for too long. The analogy is like this:\nWe are queuing to buy at a restaurant (request), but we can\u0026rsquo;t wait for the queue (timeout limit). And we decided to leave the restaurant (return error). Max lifetime Lifetime determines how long a connection can live before being reset or closed by the pool, even if the connection is active. Usually used to avoid old connection problems or update configurations to ensure connections remain fresh and reliable.\nFor example, if we set max lifetime 20 minutes, any connection that has been used for 20 minutes will be closed and removed from the pool and will be replaced by a new connection that will be created.\nThis is important because sometimes some database servers have a time limit for old connections which can prevent stale connections** problems. The analogy is like this:\nAn office has 3 security officers per shift. Each officer works a maximum of 8 hours (Max Lifetime) After 8 hours, the officers must be replaced with new officers, even though the previous 5 officers are still in good condition, there are no problems and work is running smoothly. But it prevents officers from getting tired and losing concentration when checking incoming visitors (requests). Implementation Now we will try to implement a connection pool using Golang. Note the following code:\npackage main import ( \u0026#34;database/sql\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;time\u0026#34; _ \u0026#34;github.com/go-sql-driver/mysql\u0026#34; ) func main() { dsn := \u0026#34;username:password@tcp(127.0.0.1:3306)/dbname?parseTime=true\u0026#34; db, err := sql.Open(\u0026#34;mysql\u0026#34;, dsn) if err != nil { log.Fatalf(\u0026#34;Failed to open connection to database: %v\u0026#34;, err) } defer db.Close() db.SetMaxOpenConns(10) db.SetMaxIdleConns(5) db.SetConnMaxLifetime(time.Minute * 10) db.SetConnMaxIdleTime(time.Minute * 5) err = db.Ping() if err != nil { log.Fatalf(\u0026#34;Failed to ping database: %v\u0026#34;, err) } fmt.Println(\u0026#34;Successfully connected to database!\u0026#34;) rows, err := db.Query(\u0026#34;SELECT id, name FROM users\u0026#34;) if err != nil { log.Fatalf(\u0026#34;Failed to run query: %v\u0026#34;, err) } defer rows.Close() for rows.Next() { var id int var name string if err := rows.Scan(\u0026amp;id, \u0026amp;name); err != nil { log.Fatalf(\u0026#34;Failed to read query results: %v\u0026#34;, err) } fmt.Printf(\u0026#34;ID: %d, Name: %s\\n\u0026#34;, id, name) } if err := rows.Err(); err != nil { log.Fatalf(\u0026#34;Error after iteration: %v\u0026#34;, err) } } We can focus on the following code snippet:\ndb.SetMaxOpenConns(10) db.SetMaxIdleConns(5) db.SetConnMaxLifetime(time.Minute * 10) db.SetConnMaxIdleTime(time.Minute * 5) Here\u0026rsquo;s the explanation:\nSetMaxOpenConns(10): Determines the maximum number of connections that can be open simultaneously.\nSetMaxIdleConns(5): Determines the maximum number of idle connections maintained in the pool.\nSetConnMaxLifetime(time.Minute * 10): Determines how long a connection can live before being reset or closed.\nSetConnMaxIdleTime(time.Minute * 5): Determines how long an idle connection can be maintained before being closed.\nConclusion The use of connection pooling is very important, as this can anticipate the application running smoothly when handling an erratic load of incoming requests. But the right settings must also be implemented, so that the use of connection pooling can be more optimal.\nIf you have additions or corrections to the discussion above, let\u0026rsquo;s discuss it in the comments column. Hope it helps 👋.\n","date":"2025-02-06T09:46:35+07:00","description":"What a connection pool is, why it matters for performance, and how to configure one in backend applications.","permalink":"https://rizkynotes.com/en/posts/connection-pool-in-backend-development-basic-concept-benefits-and-implementation/","tags":["go","golang","tips"],"title":"Connection Pool in Backend Development: Basic Concept, Benefits, and Implementation"},{"categories":null,"content":"Image by Hans-Peter Gauster from Unsplash Let\u0026rsquo;s say we are building an E-commerce API that will receive several orders, each order process has several statuses such as Pending, Processed, Shipped, Delivered, Cancelled. And our application receives input strings which will be stored in the database, for example the status is Processed, received Process, Processing or something else that causes data inconsistencies. Here Enum has an important role.\nIn Golang enums unlike other languages ​​such as Java or C# which offer built-in support for enums, Go takes a different approach. In Go, enums are not a native language feature, but developers have several techniques that can be used to achieve similar functionality.\nUnderstanding Enum In Golang, enums (short for enumerations) provide a way to represent a set of named constants. Although Go does not have built-in enum types like some other languages, developers can emulate enum-like behavior using constants or custom types. Let\u0026rsquo;s discuss the purpose and syntax of enums in Go:\nObjective Readability and Maintainability: Enums make code more readable and easy to understand by giving meaningful names to specific values. This increases the maintainability of the code because the purpose of each constant becomes easier to understand.\nType Safety: Enums help enforce type safety by restricting variables to a predefined set of values. This reduces the possibility of runtime errors caused by using incorrect values.\nCreate Enums Here we will discuss step by step how to create an enum in Golang, so that it is easy to understand at each stage we will explain the meaning of the code we write.\nCreate a New Type The first thing we will do is create a new type for the enum we need. The method is quite easy, we only need to use the keyword type and followed by the name of the type here with the name StatusOrder and the type here we define the type unsigned integer like this:\ntype StatusOrder uint Well, just make it easy.\nDefine constant ENUM With the new type that we have created, now is the time for us to define some of the order statuses that we have with constants. Where we define the type StatusOrder which we create as the type, like this:\nconst ( Pending StatusOrder = iota Processed Shipped Delivered Cancelled ) Maybe you ask, what is the keyword iota? This keyword makes GO assign a value of 0 to the first constant and then increase the value by 1 sequentially for each subsequent constant. This makes it easier for us rather than defining the values ​​manually 1 by 1. About iota you can read here.\nFunction Strings The next step we will take is to create a String function that is used to represent each string value from the StatusOrder enum.\nfunc (s StatusOrder) String() string { switch s { case Pending: return \u0026#34;Pending\u0026#34; case Processed: return \u0026#34;Processed\u0026#34; case Shipped: return \u0026#34;Shipped\u0026#34; case Delivered: return \u0026#34;Delivered\u0026#34; case Cancelled: return \u0026#34;Cancelled\u0026#34; default: return \u0026#34;Unknown\u0026#34; } } Does the function name have to be String? We\u0026rsquo;ll discuss it at the end.\nTesting Now we will carry out testing, the last code we will write is, function main and in it we print the results using the help of the fmt package. yes.\nfunc main() { processed := Processed fmt.Printf(\u0026#34;Order Status: %s (%d)\\n\u0026#34;, processed, processed) pending := Pending fmt.Printf(\u0026#34;Order Status: %s (%d)\\n\u0026#34;, pending, pending) } Now the complete code can be seen here.\nOf course, we carry out the last step and we will see the results like this:\nFmt Stringer The question may arise, why is the String function also called when we call the enum constant?. The answer is because we use the fmt package. The fmt package explicitly uses the fmt.Stringer interface to process types that implement the String() method. So, if you don\u0026rsquo;t use fmt, the String() method will not be called automatically. To explain more, you can explore in more detail here.\nConclusion Although Golang does not offer native enum types, the techniques we learn here are often used in building applications with Golang. And for the type itself, we can freely use other types, not just integer. By utilizing this technique, readability, ease of maintenance and security can be significantly improved.\nMaybe there are some points explained above that you feel are lacking, we can discuss them in the comments column below. Hope it helps 👋.\nReading References fmt Go Wiki: Iota Enums in Golang: Techniques, Best Practices, \u0026amp; Use Cases Mastering ENUMs in Go ","date":"2024-12-31T15:18:47+07:00","description":"How to implement clean, type-safe enums in Go, even though the language has no built-in enum support.","permalink":"https://rizkynotes.com/en/posts/mastering-enums-in-go/","tags":["go","golang","tips"],"title":"Mastering Enums in Go"},{"categories":null,"content":"In backend application development, ensuring application security and stability is an absolute must. The backend is the backbone of the application, which is responsible for handling business logic, storing data, and interacting with external systems. Writing strong and reliable code is essential for all software developers. However, no matter how careful we are, bugs and unexpected situations can still occur. This is where Defensive programming comes into play.\nDefensive programming is a coding practice aimed at ensuring that software functions correctly even when unexpected events or invalid input occur. For a backend developer defensive programming is an important approach, which allows us to design applications that can survive bad input, system errors, and external attacks.\nIn this article, we will discuss several points on how defensive programming can be applied to increase the resilience of backend applications and provide examples in the Golang language to illustrate how to implement it, although its implementation is not tied to just one language.\nValidate Input Data from External Sources External sources such as third-party API users, or databases can be entry points for various attacks, especially if the input provided is invalid or malicious. Attacks such as SQL injection, cross-site scripting (XSS), and command injection often occur due to input that is not properly filtered. We can apply the following points to overcome this:\n1. Input Sanitation Any input from outside sources must be sanitized. Input sanitization is the process of ensuring that data received from users or external sources is cleaned of malicious characters before it is used in an application, especially when the data is used for SQL queries, system commands, or output to the browser. For example, if you receive string input from a form, avoid entering that input directly into a SQL query without sanitization. What you can do is use prepared statements to prevent SQL Injection. With prepared statements, user input is not allowed to be part of the SQL command, but rather as a safe parameter. For example, we have input to check the user\u0026rsquo;s username and password for login and our backend system, rather than directly running the following query:\n// direct query with parameters query := fmt.Sprintf(\u0026#34;SELECT * FROM users WHERE username = \u0026#39;%s\u0026#39; AND password = \u0026#39;%s\u0026#39;\u0026#34;, \u0026#34;john_doe\u0026#34;, \u0026#34;12345\u0026#34;) rows, err := db.Query(query) It is better to use prepared statements, where we ensure that user input is never executed as part of a SQL query. Input is considered a value, not part of a SQL command as follows:\n// prepared statement stmt, err := db.Prepare(\u0026#34;SELECT * FROM users WHERE username = ? AND password = ?\u0026#34;) if err != nil { log.Printf(\u0026#34;Error while prepared statement: %v\u0026#34;, err) return } defer stmt.Close() // excecution prepared statement username := \u0026#34;john_doe\u0026#34; password := \u0026#34;12345\u0026#34; rows, err := stmt.Query(username, password) if err != nil { log.Printf(\u0026#34;Error while run query: %v\u0026#34;, err) return } defer rows.Close() 2. Data Type Validation Data type validation is the process of checking and ensuring that the type of data received is as expected before further processing. This is important to prevent errors in data processing and also reduce security risks, such as SQL injection and XSS. For example, if an API expects an email format, validate that the input is indeed an email format. The following is an example of carrying out the data type validation process in Golang:\nfunc main() { input := \u0026#34;hello@world.com\u0026#34; // Use regex to validate email format re := regexp.MustCompile(`^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,} $`) if !re.MatchString(input) { fmt.Println(\u0026#34;Invalid email format.\u0026#34;) return } fmt.Println(\u0026#34;Valid email:\u0026#34;, input) } Here, we use a regular expression to validate whether the input corresponds to the correct email format.\n3. Whitelist \u0026amp; Blacklist Whitelist and blacklist are two approaches used to validate and manage user input in applications. Both have the goal of improving security and preventing exploits, but they work and are applied differently.\n- whitelist Whitelisting is an approach where only certain data or characters are allowed to be processed by the application. In the context of user input, a whitelist determines what is considered \u0026ldquo;safe\u0026rdquo; and allows only input that meets those criteria. It is often used to validate very specific data. Example of use:\nfunc isValidUsername(username string) bool { // using regex to check allowed characters re := regexp.MustCompile(`^[a-zA-Z0-9_-]+$`) return re.MatchString(username) } func main() { input := \u0026#34;user_name-123\u0026#34; if isValidUsername(input) { fmt.Println(\u0026#34;Valid username:\u0026#34;, input) } else { fmt.Println(\u0026#34;Invalid username.\u0026#34;) } } In the example above, the use of regex ^[a-zA-Z0-9_-]+$ is used to limit the characters allowed in the username. Only letters, numbers, _ (underscore), and - (dash) are accepted.\n- Blacklist Blacklisting is an approach where you specify certain data or characters that are not allowed. In this case, all other inputs are considered safe, except those on the blacklist. This approach is often used when it is difficult to predict all possible safe characters. Example of use:\nfunc isValidInput(input string) bool { // List not allowed characters blacklist := []string{\u0026#34;\u0026lt;\u0026#34;, \u0026#34;\u0026gt;\u0026#34;, \u0026#34;\u0026amp;\u0026#34;} for _, char := range blacklist { if strings.Contains(input, char) { return false } } return true } func main() { input := \u0026#34;hello \u0026amp; welcome\u0026#34; if isValidInput(input) { fmt.Println(\u0026#34;Valid input:\u0026#34;, input) } else { fmt.Println(\u0026#34;Invalid input.\u0026#34;) } } Here, we use a blacklist to check whether the input contains unwanted characters. If present, the input is considered invalid.\n4. Input Limit Input Limit is the practice of limiting the amount of data or size of input that can be accepted from users in an application. The goal is to prevent excessive use of resources, improve application performance, and protect against potential attacks, such as Denial of Service (DoS) or SQL injection. The following is an example of its implementation:\nfunc validateUsername(username string) bool { if len(username) \u0026gt; 20 { return false // Username is too long username is too long } return true // username is valid } func main() { input := \u0026#34;ThisIsAReallyLongUsername\u0026#34; if validateUsername(input) { fmt.Println(\u0026#34;Valid username:\u0026#34;, input) } else { fmt.Println(\u0026#34;Invalid username: username cannot exceed 20 characters.\u0026#34;) } } Of course, some of the points above can be applied to any programming language and can be used using libraries that are widely available so that the data validation process is easier than carrying out a manual validation process like some of the examples above.\nData Security Sensitive data such as personal information or credentials must always be protected, both when stored and when transferred. We can apply the following points to overcome this:\n1. Data encryption Sensitive data such as passwords or credit card information should always be encrypted. Encryption involves the process of converting readable data (plaintext) into an unintelligible form (ciphertext) using an encryption algorithm and encryption key. The main purpose of encryption is to ensure that even if data falls into the wrong hands, it cannot be read without the right key to decrypt it.\nUse modern encryption algorithms such as AES and avoid outdated algorithms such as MD5 or SHA1.\n2. Use HTTPS Use HTTPS (Hypertext Transfer Protocol Secure) to ensure that all data transferred between the client and server is properly encrypted. HTTPS (Hypertext Transfer Protocol Secure) is a secure version of HTTP, which is a protocol used for data transfer between a browser (client) and a server.\nHTTPS works by adding a layer of security by using TLS (Transport Layer Security) or SSL (Secure Sockets Layer) to encrypt data sent over the network. This is especially important in backend development because it prevents third parties (such as hackers) from reading or modifying data that is being sent between the client and the server.\n3. Tokenization \u0026amp; Masking In certain cases, tokenization or masking can be used to reduce the risk of a data breach. Tokenization and Masking are two data security techniques frequently used in backend applications to protect sensitive data.\nTokenization is the process of replacing sensitive data (such as credit card numbers) with meaningless values ​​called “tokens.” The original data is stored separately and can only be linked back to the token through a system that has access to the system that manages the token. This is important because if a data leak occurs, but the leaked data is only a token and not real data, the risk of attack is greatly reduced because the token has no value outside the system context. For example, in online payments, credit card numbers can be replaced with tokens so that card data does not actually need to be stored on the server.\nData masking is the process of hiding part of sensitive data so that only some parts are visible. This is often used in user interfaces or reports to prevent complete information from being exposed. How does this work? data masking protects sensitive information when displayed to users who do not need to know the full data. Even if the data displayed in the UI or logs is stolen, the masked data only shows partial information, so the risk is lower. An example that we may often see is hiding part of a credit card number in a display like this: **** **** **** 1234.\nError Management \u0026amp; Logging In building a system errors will always occur, but how we handle them determines whether our application can survive without causing damage or loss of data. Good logging helps track problems in production systems and detect suspicious patterns. Here are some points that we do:\n1. Handle Errors Appropriately Don\u0026rsquo;t just catch errors without taking action or providing information. It\u0026rsquo;s best to log errors and provide a good fallback to avoid a total crash. For example, if the connection to the database fails, try retrying or switching to read-only mode temporarily.\n2. Granular Logging Log critical events such as API requests, authentication failures, or database connection problems with a sufficient level of detail. However, remember to avoid recording sensitive information such as passwords in logs.\n3. Log Splitting Use different log levels such as INFO, WARNING, and ERROR to separate different types of events. More granular logs make it easier for developers to diagnose problems.\nFor a more complete discussion and examples of Error Management \u0026amp; Logging I have written an article about this here: Observability - Why logging is important\nAvoid mistakes elegantly Crashing is a response that occurs when the system encounters an unexpected error. However, errors can be avoided or minimized through better handling. The strategies below can help us handle errors elegantly to make the system more resilient.\n1. Timeouts When backend applications access external services or wait for resources such as databases, it is important to use timeouts. Timeouts prevent code from being \u0026ldquo;stuck\u0026rdquo; in an indefinite waiting state, thereby maintaining application performance and avoiding deadlocks. The following is an example of implementing a timeout when accessing an external service:\nclient := http.Client{ Timeout: 5 * time.Second, // Set timeout 5 detik } resp, err := client.Get(\u0026#34;https://example.com/api\u0026#34;) if err != nil { fmt.Println(\u0026#34;Request failed:\u0026#34;, err) return } defer resp.Body.Close() fmt.Println(\u0026#34;Response received:\u0026#34;, resp.Status) In this example, we set a 5 second timeout for HTTP requests. If the service doesn\u0026rsquo;t respond within 5 seconds, the app won\u0026rsquo;t hang, and we can handle the error by announcing the failure.\n2. Resource Existence Checks Before accessing other resources such as databases, files or APIs, always verify their existence and availability. This prevents applications from making requests to resources that may not exist, which saves time and avoids unnecessary errors. Here\u0026rsquo;s an example of how we check:\nfunc main() { filePath := \u0026#34;./data.txt\u0026#34; _, err := os.Stat(filePath) // check status file if os.IsNotExist(err) { fmt.Printf(\u0026#34;File %s not found.\\n\u0026#34;, filePath) } else { fmt.Printf(\u0026#34;File %s is available.\\n\u0026#34;, filePath) } } 3. Fallback When the system carries out certain processes, we might get an error. For example, our system sends an OTP during the authentication process. When the OTP sending process fails or gets an error, we can use a fallback mechanism or backup method, such as diverting OTP sending to our backup provider.\nBy implementing the strategies above, we can reduce the frequency of errors that cause crashes, maintain application performance, and improve the overall user experience.\nSecure and Solid API Design We can see that APIs function as the backbone for applications or web that require dynamic data resources. However, because they are considered such a critical backbone, APIs are often the main target for attacks, so it is important to ensure APIs that are secure and difficult to exploit.\n1. Authentication \u0026amp; Authorization Use strong authentication (such as OAuth or JWT) to ensure that only authorized users or systems can access our APIs. Implement granular access control to limit the access rights of clients who access the API that we have. This can prevent clients with malicious intent from accessing the resources we have.\n2. Rate Limiting and Throttling Rate Limiting and Throttling are critical to controlling API usage and preventing misuse or overloading. The application of rate limiting is also to prevent attacks such as DDoS or excessive exploitation of the API by users or bots, and also to maintain the stability and availability of the API for all users. Rate Limiting works by regulating the volume of requests from API clients within set intervals, ensuring fair and equal access to the resources we have.\nEfficient Use of Resources Backend systems often have to manage various resources such as database connections, files, or memory. Efficient resource management is critical to maintaining application stability and performance. By handling resources wisely, we can avoid problems such as \u0026ldquo;resource leaks\u0026rdquo; and system failures. The following are several strategies that can be used for efficient use of resources:\n1. Connection Pooling Connection Pooling is a technique where a number of connections to a service such as a database are recycled rather than creating a new connection every time an application needs access. With connection pooling, applications do not need to create expensive new connections every time there is a request, but instead use existing connections in the \u0026ldquo;pool\u0026rdquo;.\nWhy is connection pooling important?, Every connection to a database or other service has overhead in terms of time and memory. Continuously opening and closing connections can drain system resources. By recycling existing connections, applications can respond more quickly and reduce the load on external services such as databases.\n2. File \u0026amp; Memory Management When dealing with files or memory allocation, it is important to ensure that resources are released after they have been used. If not, it could cause resource leaks which result in the application running out of memory or file handles.\nFor example, in Golang we can use the built-in function defer to ensure files, connections, or other resources are always released after they are finished using them. Then also check for errors (error handling) when working with resources, so that if a problem occurs, the resource can still be released correctly.\nHere\u0026rsquo;s a little code snippet of how we use defer to turn off an operation and close it after a function has finished carrying out a process:\nfunc main() { // open file process file, err := os.Open(\u0026#34;example.txt\u0026#34;) if err != nil { fmt.Println(\u0026#34;Error opening file:\u0026#34;, err) return } // Make sure the file is closed when finishedinished defer file.Close() // Process file... } By using defer, we ensure that the file will be closed as soon as the function exits, whether it is successful or an error occurs. This helps prevent file handle leaks that can occur if you forget to close a file.\n3. Asynchronous Operations In situations where the application performs heavy tasks or slow I/O operations (such as accessing large files, or calling external APIs), running it Asynchronously is an efficient way to avoid blocking the application. Where applications that perform synchronous operations can be hampered while waiting for heavy tasks to complete, which will affect overall performance. So the use of asynchronous processes is sometimes important because it allows applications to continue other tasks while waiting for I/O or network tasks to complete.\nLet\u0026rsquo;s say we have a case where we clear the Redis cache after data input. The cache deletion process can be run asynchronously so as not to slow down the main process. This means the application does not need to wait for Redis to respond before continuing to execute the next code. As a result, the application remains responsive and efficient.\nConclusion As a backend developer, defensive programming is an essential approach to ensuring that the applications we build can handle errors, invalid input, and external threats properly. By practicing the techniques above, we can ensure that the backend applications we build are more secure, stable and resilient in facing various unexpected situations. Defensive programming helps us create systems that not only function well, but are also able to survive the worst conditions.\nOf course, there may be many other important points that are not discussed above, you can add or provide suggestions in the comments column above.\nHopefully it\u0026rsquo;s useful 👋.\nReference The Code Knight: Mastering the Craft of Defensive Programming Defensive Programming in C#: Best Practices and Examples The Art of Defensive Programming ","date":"2024-11-03T11:42:30+07:00","description":"Defensive programming techniques for backend developers — input validation, error handling, and building robust, secure systems.","permalink":"https://rizkynotes.com/en/posts/defensive-programming-as-a-backend-developer-building-robust-and-secure-systems/","tags":["tips","backend engineering"],"title":"Defensive Programming as a Backend Developer: Building Robust and Secure Systems"},{"categories":null,"content":"When your application starts on a large scale, the increase in users will increase. What is very likely to happen is that the user\u0026rsquo;s location is not only in the same area, it could be in another area that has a different time zone. So as a Backend developer, things related to handling time zone differences are very important to think about.\nI recently came across an issue involving time zones. Let\u0026rsquo;s be honest, dealing with dates and times is one of the most complicated areas a human being has to deal with. And this was an opportunity for me to learn how to properly handle dates and times on the server side.\nIn this article, I will share my experience on how I handle time zone differences on the server side as a Backend developer. Maybe if someone is willing to correct and provide additional input that would be valuable for me.\nAbout Time Zone Time zones are a standard time division system used throughout the world to organize and standardize the measurement of time. This concept emerged as a response to the need for global time coordination, especially along with developments in communications and transportation technology.\nThe basic principle of time zones is the division of the earth into 24 zones. Each time zone is generally one hour different from the zone next to it. The main reference for time zones is Greenwich Mean Time (GMT) or Coordinated Universal Time (UTC), which is at the zero degrees longitude line that passes through Greenwich, England.\nIllustration by Hellerick from Wikimedia Commons\nA small example is when the clock shows 12:00 noon in Jakarta, in New York the time shows 00:00 or midnight. This means that while Jakartans are enjoying lunch, New Yorkers are just starting their new day. From here you can certainly imagine the importance of handling time zones correctly in building an application.\nTimezone handling on the server side After we have seen the explanation above, now we will go into the points that can be done when our server application receives a request from a client that accesses our API to handle its time zone.\nIn this article, I will discuss several approaches to handling time zones on the server side. Here I will use code examples in the Golang (Go) language. Golang has a time package for working with time-related data which is considered quite complete. Here are some points that we will discuss:\nHow to save date to DB Conversion of user\u0026rsquo;s local time Testing and Validation How to save date to DB The first thing we will discuss is which time we will save in the database, for example we have an ecommerce application that does flash sales, where our application is already on an international scale.\nWhen a user processes a purchase transaction in America or if the user is in Indonesia, the user will send their different local time to the server. The question is, will our database store time data according to the user\u0026rsquo;s local time? If the answer is yes, it is likely a complicated problem when we want to retrieve the data or for example the admin wants to do data processing, which user makes the earliest transaction.\nTo overcome this, the best practice is to store transaction times in the UTC (Coordinated Universal Time) time zone which is the primary time standard for clocks and time settings. Here is the application of the time to UTC.\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;time\u0026#34; ) func main() { now := time.Now() fmt.Printf(\u0026#34;Local time: %s\\n\u0026#34;, now) nowInUTC := now.UTC() fmt.Printf(\u0026#34;UTC time: %s\\n\u0026#34;, nowInUTC) } Let\u0026rsquo;s see the meaning of the code above.\nFirst, in the now := time.Now() code line, this line uses the Now() function from the time package to get the current time based on the system\u0026rsquo;s local time zone. The result is stored in the current variable.\nThen, in the nowInUTC := now.UTC() line, here we convert the local time (now) to UTC time using the UTC() method. The results are stored in a new variable nowInUTC and this time can be stored on the server, where it is hoped that developers can avoid ambiguity and errors that may arise due to time zone differences between servers and users in various regions with different time zones.\nHere are the results if we run the code above:\nBut this is not always the best solution that you should use. There may be some points you can keep in mind in certain use cases, one of which is is it true that our users come from different time zones? If it\u0026rsquo;s not possible, perhaps storing the time in UTC will add complexity to your code.\nChange time to user locale At the discussion point above, we have agreed to store user time data in one location, namely UTC. Now how users can see accurate time according to their location. An example of the discussion above is a flash sale on an e-commerce application that we have, where users also want to know information about which user made the first transaction. So at this point, converting the time we store in the database to the user\u0026rsquo;s local time is another important thing that we should not ignore.\nThe approach I take is that the server side always asks the client to send the timezone on the user side. This can be done on the request side where the client sends a header with the key timezone and has the user\u0026rsquo;s timezone value. For example, Indonesia has 3 time zone divisions, namely WIT(+9), WITA(+8), WIB(+7). Where each zone has a difference of 1 hour. If previously on our server we stored UTC time at 00.00, then on the WIT side it was at 09.00, then on the WITA side it was at 08.00 and WIB at 07.00.\nHere\u0026rsquo;s an example code to handle the above case:\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;time\u0026#34; ) func ParseTimezoneFromString(tz string) *time.Location { if len(tz) \u0026gt; 0 { t, err := time.Parse(\u0026#34;2006 -07:00\u0026#34;, fmt.Sprintf(\u0026#34;2007 %s\u0026#34;, tz)) if err != nil { panic(err) } else { return t.Location() } } return time.Now().Location() } func main() { timeServerInUTC := \u0026#34;2024-08-04 00:00:00\u0026#34; nowInUTC, err := time.Parse(\u0026#34;2006-01-02 15:04:05\u0026#34;, timeServerInUTC) if err != nil { panic(err) } fmt.Printf(\u0026#34;UTC time: %s\\n\u0026#34;, nowInUTC) witLocation := ParseTimezoneFromString(\u0026#34;+09:00\u0026#34;) nowInWIT := nowInUTC.In(witLocation) fmt.Printf(\u0026#34;WIT time: %s\\n\u0026#34;, nowInWIT) witaLocation := ParseTimezoneFromString(\u0026#34;+08:00\u0026#34;) nowInWITA := nowInUTC.In(witaLocation) fmt.Printf(\u0026#34;WITA time: %s\\n\u0026#34;, nowInWITA) wibLocation := ParseTimezoneFromString(\u0026#34;+07:00\u0026#34;) nowInWIB := nowInUTC.In(wibLocation) fmt.Printf(\u0026#34;WIB time: %s\\n\u0026#34;, nowInWIB) } credit to dikac for create this function ParseTimezoneFromString\nLet\u0026rsquo;s understand the meaning of the code above:\nFirst, we create a function ParseTimezoneFromString, where this function is used to find the time location based on the argument tz or timezone of the given user location. If the string value tz is valid, we will convert the string\u0026rsquo;s timezone using the time.Parse function to convert the string to a time object, then extract the location (timezone) from that object. And we also handle if the string is empty or parsing fails, the function returns the local time zone of the system.\nfunc ParseTimezoneFromString(tz string) *time.Location { if len(tz) \u0026gt; 0 { t, err := time.Parse(\u0026#34;2006 -07:00\u0026#34;, fmt.Sprintf(\u0026#34;2007 %s\u0026#34;, tz)) if err != nil { panic(err) } else { return t.Location() } } return time.Now().Location() } Next we also define time data in the following string format:\ntimeServerInUTC := \u0026#34;2024-08-04 00:00:00\u0026#34; nowInUTC, err := time.Parse(\u0026#34;2006-01-02 15:04:05\u0026#34;, timeServerInUTC) if err != nil { panic(err) } You can think of this as the timing data that we get from the server. And parse it into a time object.\nNext, we try to find the user\u0026rsquo;s accurate location based on the ParseTimezoneFromString function that we previously created based on the string argument that we defined. What needs to be paid attention to is that this string argument is what is meant by the value of the timezone header sent by the client via the incoming request.\nWe can use the location we get from the ParseTimezoneFromString function to convert or shift the time we get from the server to the user\u0026rsquo;s local time. We can do this using the In function which is also included in the time package which we can see in the following code snippet:\nnowInWIT := nowInUTC.In(witLocation) nowInWITA := nowInUTC.In(witaLocation) nowInWIB := nowInUTC.In(wibLocation) If we run it, we will get the time that corresponds to the timezone location that we defined.\nTesting and Validation The final point is no less important, namely testing and validation. When the development process often causes developers to make unexpected mistakes, testing and validation are always important.\nIn the discussion of point 2 above, the ParseTimezoneFromString function has been important in handling our time zones. Repeated testing and validation are important to make our applications get results that meet our expectations.\nFor testing, it is recommended to use unit tests, where testing will be carried out on the smallest unit with several scenarios that can be added. The more scenarios there are, the less likely it is to handle these time differences.\nConclusion Handling time zones can indeed be tricky for backend developers. However, it\u0026rsquo;s important to remember that every challenging task we overcome contributes to our growth and skill improvement. Properly managing time zones is not just a technical necessity, it ensures accuracy in scheduling and provides a smooth experience for our application users across different regions.\nThe points shared in this article about storing times in UTC, converting to user local times, and implementing robust conversion functions are starting points in tackling this complex issue. However, I acknowledge that there may be shortcomings or areas for improvement in the approaches discussed. This is why additional input and suggestions from the developer community are invaluable.\nI sincerely hope that the insights and code examples provided in this article will be helpful to you in the future when you encounter time zone-related challenges in your projects. Remember, the goal is to create applications that work seamlessly for users, regardless of their geographical location.\nLet\u0026rsquo;s continue this discussion in the comments section below. I\u0026rsquo;d love to hear about your experiences with handling time zones, any challenges you\u0026rsquo;ve faced, or alternative approaches you\u0026rsquo;ve found effective. Your insights could be extremely valuable to me and other readers facing similar issues.\nThank you for reading, and I hope this article proves useful in your development journey. Let\u0026rsquo;s keep learning and improving together! 👋\nReading References Time zone What Is a Time Zone? How to Handle Timezones and Synchronize Your Software with International Customers Dealing with timezones in web development Working with Dates and Times in Go: Handling Timezones and Formatting ","date":"2024-10-01T09:36:41+07:00","description":"How to correctly handle time zones and keep timestamps consistent on the server side using Go.","permalink":"https://rizkynotes.com/en/posts/how-to-handle-time-zones-and-sync-your-software-on-the-server-side-using-go/","tags":["go","golang","tips"],"title":"How to Handle Time Zones and Sync Your Software on the Server Side Using Go"},{"categories":null,"content":"In an increasingly complex digital era, observability is the main key in managing modern software systems. One of the most important pillars of observability is logging. Let\u0026rsquo;s explore why logging is so important and how to make optimal use of it.\nWhat is Logging? Logging is the process of recording activities and events in a system. This includes a variety of information, from error messages, user activity, to system performance. Think of logging as an airplane \u0026lsquo;black box\u0026rsquo; for your system - always recording what\u0026rsquo;s happening, ready to provide insights when needed.\nWhy is Logging So Important? Here are some points that can be considered why logs are important:\nFaster Problem Solving With good logs, development teams can identify root causes without guesswork. It\u0026rsquo;s like having a treasure map when looking for bugs!\nSecurity Improvements Logs can be your \u0026lsquo;spy\u0026rsquo; in detecting suspicious activity. Security teams can respond to threats more quickly, such as having a fire department always on standby.\nPerformance Analysis Through logs, you can identify bottlenecks in the system. It\u0026rsquo;s like having a personal doctor for your app\u0026rsquo;s health.\nUnderstanding User Behavior User activity logs provide valuable insight into how the product is used. It\u0026rsquo;s like having a personal assistant constantly observing and reporting customer preferences.\nBest Practices in Logging To maximize the benefits of logging, below are some of the best practices that can be carried out:\nDetermine the Appropriate Log Level Using these appropriate log levels can help you filter information quickly, such as sorting logs by urgency.\nThe following is an example of displaying logs using the Golang language with various levels. Here we use the Logrus.\npackage main import ( \u0026#34;github.com/sirupsen/logrus\u0026#34; ) func main() { log := logrus.New() log.SetLevel(logrus.DebugLevel) log.Debug(\u0026#34;Starting app..\u0026#34;) log.Info(\u0026#34;User has successfully logged in\u0026#34;) log.Warn(\u0026#34;CPU usage exceeds 80%\u0026#34;) log.Error(\u0026#34;Failed to save data to database\u0026#34;) log.Fatal(\u0026#34;A critical error occurs, the application will stop\u0026#34;) } The following is an explanation for the several log levels above:\nDEBUG: Detailed information for debugging, usually only enabled during development. INFO: General information about the normal flow of the application. WARNING: For situations that have the potential to become problematic in the future, but do not stop the application. ERROR: An error that causes a specific function to fail, but the application is still running. FATAL: Serious error that may cause the application to stop. Include relevant contextual information Each log entry should provide enough context to understand what happened. This could include:\nTimestamp. Transaction or session ID. User ID (if relevant). Function or module name. Relevant input data (be careful with sensitive data). Stack trace for errors This is an example of code when printing a log, including context information that will help us trace.\npackage main import ( \u0026#34;github.com/sirupsen/logrus\u0026#34; \u0026#34;time\u0026#34; ) type UserAction struct { UserID int Action string Timestamp time.Time } func main() { log := logrus.New() log.SetLevel(logrus.DebugLevel) // Use format json log.SetFormatter(\u0026amp;logrus.JSONFormatter{}) // Dummy data action := UserAction{ UserID: 12345, Action: \u0026#34;checkout\u0026#34;, Timestamp: time.Now(), } // Print log log.WithFields(logrus.Fields{ \u0026#34;user_id\u0026#34;: action.UserID, \u0026#34;action\u0026#34;: action.Action, \u0026#34;timestamp\u0026#34;: time.Now().Format(time.RFC3339), \u0026#34;session_id\u0026#34;: generateSessionID(), \u0026#34;module\u0026#34;: \u0026#34;payment_processor\u0026#34;, \u0026#34;ip_address\u0026#34;: \u0026#34;192.168.1.100\u0026#34;, }).Error(\u0026#34;Payment failed\u0026#34;) } func generateSessionID() string { return \u0026#34;sess_abc123\u0026#34; } We can see that we have included several elements of context information that can make it easier for us to carry out tracing in the future. What are the conveniences in question, namely that we can search logs based on level, for example the error level in the code example above, and also based on time and others based on the information we enter.\nUse consistent formatting A consistent log format makes parsing and analysis easier, especially if using automated tools (regarding tools, will be discussed below). Formatting also makes it easier for us to search logs based on criteria, for example log level, message, or time. Example format:\n[TIMESTAMP] [LEVEL] [MODULE] [MESSAGE] Or JSON format for easy parsing like the results in the code example above:\n{ \u0026#34;action\u0026#34;: \u0026#34;checkout\u0026#34;, \u0026#34;ip_address\u0026#34;: \u0026#34;192.168.1.100\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;error\u0026#34;, \u0026#34;module\u0026#34;: \u0026#34;payment_processor\u0026#34;, \u0026#34;msg\u0026#34;: \u0026#34;Payment failed\u0026#34;, \u0026#34;session_id\u0026#34;: \u0026#34;sess_abc123\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-06-26T20:59:02+07:00\u0026#34;, \u0026#34;timestamp\u0026#34;: \u0026#34;2024-06-26T20:59:02+07:00\u0026#34;, \u0026#34;user_id\u0026#34;: 12345 } Implement log rotation to manage file size Log rotation prevents log files from becoming too large and difficult to manage. This involves:\nLimits the size of log files.\nCreate new log files periodically (e.g. daily or weekly).\nArchive or delete old log files.\nUsing tools such as logrotate on Linux or a logging framework that supports rotation.\nConsider privacy and security in logged information Security and privacy are very important in logging:\nDo not log sensitive data such as passwords or credit card information.\nMask or encrypt personal data if necessary.\nEnsure access to log files is restricted to authorized personnel only.\nImplement a retention policy to delete old logs according to company policies and regulations.\nTools for Monitoring and Analyzing Logs As system complexity increases, the need for sophisticated tools to monitor and analyze logs also becomes increasingly important. Here are some popular tools that can help with observability and log analysis:\nGrafana Grafana is an open-source platform for visualizing our log data. These tools can be integrated into various data sources including logs. Enables the creation of customized and interactive dashboards. Suitable for real-time visualization of metrics and logs.\nNew Relic New Relic is an all-in-one observability platform Provides log analysis, tracing, and metrics in one place. There are also AI features to detect anomalies and correlate problems. Suitable for monitoring large-scale applications and infrastructure.\nLoki Loki is a lightweight and cost-effective log aggregation system. Loki is designed to work well with Grafana Uses label-based indexes, similar to Prometheus Ideal for organizations already using Prometheus and Grafana.\nAWS CloudWatch Logs Insights This integrated log analysis service from AWS enables querying and analysis of logs from various AWS services. Feature to detect slow queries in RDS and other database services Easy integration with other AWS services.\nConclusion Logging is not just an additional feature, but a vital component in building a reliable system. With proper implementation, logging can become your supersensor - providing full visibility into system operations, helping prevent problems before they occur, and speeding resolution when problems arise.\nSo, start investing in good logging practices today. Remember, in the world of complex technology, good logs can be a guiding light in the midst of a storm!\nIf you have additional information, please enter it in the comments column below.\nReading References Github Repository\nApplication Logging and its importance\nWhy is Log Management Important?\n10 Observability Tools in 2024: Features, Market Share and Choose the Right One for You\nTop 20 Best Log Analysis Tools and Log Analyzer (Pros and Cons)\n","date":"2024-07-28T09:06:36+07:00","description":"What logging is, why it matters for observability, and how to do it well in modern backend systems.","permalink":"https://rizkynotes.com/en/posts/observability-why-logging-is-important/","tags":["tips","backend engineering"],"title":"Observability: Why Logging Is Important"},{"categories":null,"content":"Introduction In today\u0026rsquo;s data-driven world, relational databases are the backbone of countless applications. They store and manage critical information, but their performance can significantly impact user experience and overall system efficiency.\nThis blog post dives into key strategies for optimizing relational databases and ensuring they run at peak performance.\n🔍 Common Query Mistake That Lead Bottleneck Before optimizing, in this writing the discussion only covers Unnecessary Full Table Scans, Inefficient Queries, Denormalization, Insufficient Hardware Resources. It\u0026rsquo;s crucial to identify performance bottlenecks. Here are some common causes:\n1. Unnecessary Full Table Scans For example, I have table products with a total 4000000 rows of data. If your query doesn\u0026rsquo;t have proper indexing, the database may be forced to scan the entire table for each lookup. For example how we can know a table caught the table scan:\nHere\u0026rsquo;s, I can try to select the products with name condition\nSELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34;; And this result I have a total time 0.628s, it\u0026rsquo;s not a long time.\nBut the surprising thing is when you run explain to see what happens:\nexplain SELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34;; See you have scanned a total of 3052749 rows just to find 1 row of data. And look at the type column here it has the value ALL, meaning you are scanning row by row looking for data. This will be a nightmare when the total data keeps getting bigger.\n2. Inefficient Queries Poorly written SQL queries can take longer to execute than necessary. Here\u0026rsquo;s an example of a poorly written SQL query:\nSELECT * FROM orders WHERE YEAR(order_date) = 2024; This query retrieves all orders from 2024. However, using the YEAR() function in the WHERE clause makes it inefficient because it prevents the query from using an index on the Order Date column.\nInstead, this forces the database to apply the YEAR() function to every row in the Orders table, potentially resulting in a full table scan. For example, the above is a query snippet from the orders table, now let\u0026rsquo;s look at the table.\nMy orders has an index on the order_date column. Run this query to see a list of available indexes:\nshow indexes from order_table; And see I do have an index for the order_date column.\nNow, when you run explain to see what happens:\nexplain SELECT * FROM orders WHERE YEAR(order_date) = 2024; It can be seen in the results below, using functions such as YEAR(), can cause the available indexes to not be used.\n3. Denormalization (Careful Approach) In some situations, denormalizing your database schema (introducing controlled redundancy) can improve query performance by reducing the need for complex joins. Here\u0026rsquo;s a simple example to illustrate denormalization in MySQL:\nLet\u0026rsquo;s consider a hypothetical scenario where you have two tables: users and orders. Below are the Original Normalized Tables:\nusers table with structure: user_id (Primary Key) username email orders table with structure: order_id (Primary Key) user_id (Foreign Key referencing user_id in the users table) order_date total_amount Now, let\u0026rsquo;s say you frequently need to retrieve a user\u0026rsquo;s orders along with their username and email. The most common way is to always join the orders table to the users table. Here\u0026rsquo;s an example query:\nSELECT orders.order_id, users.name, users.email, orders.order_date, orders.total_amount FROM orders JOIN users ON orders.user_id = users.user_id; Joining these tables every time you need this information could become a performance bottleneck, especially if the tables are large.\nWe will discuss how to denormalize it in the solutions chapter below.\n4. Insufficient Hardware Resources Database performance is highly dependent on factors such as CPU, RAM, and storage capacity. This is very difficult to see without the help of supporting software. We will discuss several solutions below.\n🎯 Solution Once you\u0026rsquo;ve identified the bottlenecks, you can employ various techniques to streamline your database:\n1. Unnecessary Full Table Scans Solution Indexes act like shortcuts for the database, allowing it to quickly locate specific data. Strategically creating and maintaining indexes on frequently used columns can significantly improve query speed.\nDiving Deeper into Indexing Types\nIn the field of database management and optimization, indexing plays an important role in improving query performance. Understanding the different types of indexing is critical to efficient data retrieval and manipulation.\nLet\u0026rsquo;s dive deeper into two fundamental types of indexing: single index and compound index, and explore their function with illustrative examples.\nSingle Index: A single index is a straightforward mechanism in which an index is created on a single column of a database table. It speeds up the search process by organizing data in indexed columns in a structured format, often a B-tree or hash table, allowing for faster retrieval of specific records.\nIn the example above, we are looking for a name in the products table with a large amount of data in the user table. Remember, to search for 1 row of data, we are required to scan each row. Now let\u0026rsquo;s try adding a single index to the name column with the following query:\nCREATE INDEX product_name_idx ON products (name); To see if an index has been created, run this query:\nshow indexes from users; Yeah, index has created:\nNow, try again to select the user with name condition:\nSELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34;; And it\u0026rsquo;s amazing how quickly you can get data when you only take 2 second to do it, compared to the previous time of 0.628s seconds.\nAnd the important thing is that we noticed many rows were scanned. Please rerun this query again.\nexplain SELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34;; There is only 1 row scanned and you can also observe that we are using an index. Small notes, if you want to remove index can run query:\nDROP INDEX product_name_idx ON product; Composite Index: Unlike single index, a composite index involves indexing multiple columns within a database table. This type of indexing is particularly advantageous when queries involve conditions on multiple columns or require sorting based on a combination of fields.\nExpanding on the previous example, let\u0026rsquo;s say we frequently run a query to retrieve products based on their name, category, and price.\nIn such a scenario, creating a combined index on the name, category, and price columns will optimize the search process by storing and sorting the data based on these combined criteria.\nFor example, if we want to search for users with a specific name, category (gadget), and price greater than 400.000, the query would look like this:\nSELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34; AND category = \u0026#34;gadget\u0026#34; and price \u0026gt; 400; As a result we get 25 data.\nSee the single index that we previously had has no effect on queries with multiple conditions.\nIf we explain with the single index we previously created, we scan 28 rows:\nexplain SELECT * from products WHERE name = \u0026#34;Produk 300665\u0026#34; AND category = \u0026#34;gadget\u0026#34; and price \u0026gt; 400; Obviously 25 lines are obtained but by scanning only 28 lines it is not a big problem. But what we need to remember is what happens if we have a large amount of data. Certainly the number of rows we scan will be much larger than the data we get. And now we try to create a composite index, by combining several columns in 1 index, a query like this:\nCREATE INDEX product_name_category_price_idx ON products (name, category, price); Now looks, a composite index has created:\nTo differentiate between single index and composite index, refer to the column key_name. A composite index is an index with multiple column_name values that share the same name as another index.\nAnd now, we trying run explain query in above for see different result:\nVoila, we only scanned 25 rows for 25 results using the composite index. And the composite index that we have is also included in the list of indexes that we will use.\n2. Inefficient Queries Solution Writing clean and optimized SQL queries is very important. Techniques such as avoiding complex functions in the WHERE clause and using appropriate JOINs can make a big difference. Or as in the example above, use the YEAR function to sort the years.\nA more efficient version of this query would directly compare the order_date column to a date range in 2024:\nSELECT * FROM Orders WHERE order_date \u0026gt;= \u0026#39;2024-01-01\u0026#39; AND order_date \u0026lt; \u0026#39;2025-01-01\u0026#39;; See nothing happens and nothing seems strange. But let\u0026rsquo;s try running an explain query:\nEXPLAIN SELECT * FROM Orders WHERE order_date \u0026gt;= \u0026#39;2024-01-01\u0026#39; AND order_date \u0026lt; \u0026#39;2025-01-01\u0026#39;; Now the index is used, and this is very important when the data we are scanning is very large.\n3. Denormalization Solution To optimize this scenario using denormalization, you might add redundant columns from the users table into the orders table:\nDenormalized Orders Table:\norders table:\norder_id (Primary Key) user_id (Foreign Key referencing user_id in the users table) order_date total_amount name (Redundant column from the users table) email (Redundant column from the users table) By adding name and email columns to the orders table, you eliminate the need for frequent joins when querying user-specific order data. This can significantly improve the performance of queries that retrieve orders along with user information.\nNow the query we have only deals with the orders table, without joining the users table and has the same results.\nSELECT orders.order_id, orders.name, orders.email, orders.order_date, orders.total_amount FROM orders; However, it\u0026rsquo;s important to note that denormalization comes with trade-offs.\nIt can lead to data redundancy, increased storage requirements, and potential data inconsistency if updates are not properly managed. Therefore, denormalization should be used judiciously and in situations where the performance benefits outweigh the drawbacks.\nAdditionally, denormalized data should be kept consistent through proper data management practices such as triggers or application logic.\n4. Insufficient Hardware Resources To monitor Insufficient Hardware Resources performance, consider using a third-party monitoring tool designed specifically for MySQL performance monitoring.\nTools such as MySQL Enterprise Monitor, Percona Monitoring and Management (PMM), New Relic, or open source solutions such as Prometheus with MySQL Exporter can provide comprehensive monitoring capabilities.\n📋 Conclusion Optimizing relational databases is critical to ensuring optimal performance and providing a smooth user experience. By overcoming common bottlenecks such as unnecessary full table scans, inefficient queries, denormalization trade-offs, and insufficient hardware resources, you can significantly improve the speed and efficiency of your MySQL database.\nKey strategies include intelligent indexing techniques such as single and composite indexes, writing clean and optimal SQL queries, and judicious application of denormalization when the performance benefits outweigh the disadvantages. Additionally, investing in adequate hardware resources and utilizing monitoring tools can help identify and resolve performance issues proactively.\nBy implementing these optimization strategies, you can unlock the full potential of your relational database, ensuring lightning-fast data retrieval, efficient data manipulation, and a responsive application experience for your users.\nThank you, hope it helps. 👋\nReading References Single vs Composite Indexes in Relational Databases Top 11 MYSQL monitoring tools in 2024 [open-source included] denormalization ","date":"2024-05-07T10:12:24+07:00","description":"Practical strategies for optimizing relational databases in MySQL — from fixing slow queries and indexing to schema design and caching.","permalink":"https://rizkynotes.com/en/posts/optimizing-relational-databases-for-best-performance-in-mysql/","tags":["database","backend engineering"],"title":"Optimizing Relational Databases for Best Performance in MySQL"}]