Codeigniter with redis and Memcache Session Drivers

 Ref: https://codeigniter.com/userguide3/libraries/sessions.html

Session Drivers

As already mentioned, the Session library comes with 4 drivers, or storage engines, that you can use:

• files
• database
• redis
• memcached

By default, the Files Driver will be used when a session is initialized, because it is the most safe choice and is expected to work everywhere (virtually every environment has a file system).

However, any other driver may be selected via the $config['sess_driver'] line in your application/config/config.php file, if you chose to do so. Have it in mind though, every driver has different caveats, so be sure to get yourself familiar with them (below) before you make that choice.

In addition, you may also create and use Custom Drivers, if the ones provided by default don’t satisfy your use case.

Redis Driver

Note

Since Redis doesn’t have a locking mechanism exposed, locks for this driver are emulated by a separate value that is kept for up to 300 seconds.

Redis is a storage engine typically used for caching and popular because of its high performance, which is also probably your reason to use the ‘redis’ session driver.

The downside is that it is not as ubiquitous as relational databases and requires the phpredis PHP extension to be installed on your system, and that one doesn’t come bundled with PHP. Chances are, you’re only be using the ‘redis’ driver only if you’re already both familiar with Redis and using it for other purposes.

Just as with the ‘files’ and ‘database’ drivers, you must also configure the storage location for your sessions via the $config['sess_save_path'] setting. The format here is a bit different and complicated at the same time. It is best explained by the phpredis extension’s README file, so we’ll simply link you to it:

https://github.com/phpredis/phpredis#php-session-handler

Warning

CodeIgniter’s Session library does NOT use the actual ‘redis’ session.save_handler. Take note only of the path format in the link above.

For the most common case however, a simple host:port pair should be sufficient:

$config['sess_driver'] = 'redis';
$config['sess_save_path'] = 'tcp://localhost:6379';

Memcached Driver

Note

Since Memcache doesn’t have a locking mechanism exposed, locks for this driver are emulated by a separate value that is kept for up to 300 seconds.

The ‘memcached’ driver is very similar to the ‘redis’ one in all of its properties, except perhaps for availability, because PHP’s Memcached extension is distributed via PECL and some Linux distrubutions make it available as an easy to install package.

Other than that, and without any intentional bias towards Redis, there’s not much different to be said about Memcached - it is also a popular product that is usually used for caching and famed for its speed.

However, it is worth noting that the only guarantee given by Memcached is that setting value X to expire after Y seconds will result in it being deleted after Y seconds have passed (but not necessarily that it won’t expire earlier than that time). This happens very rarely, but should be considered as it may result in loss of sessions.

The $config['sess_save_path'] format is fairly straightforward here, being just a host:port pair:

$config['sess_driver'] = 'memcached';
$config['sess_save_path'] = 'localhost:11211';

Bonus Tip

Multi-server configuration with an optional weight parameter as the third colon-separated (:weight) value is also supported, but we have to note that we haven’t tested if that is reliable.

If you want to experiment with this feature (on your own risk), simply separate the multiple server paths with commas:

// localhost will be given higher priority (5) here,
// compared to 192.0.2.1 with a weight of 1.
$config['sess_save_path'] = 'localhost:11211:5,192.0.2.1:11211:1';

Using memcached as a session storage with CodeIgniter

Ref: https://stackoverflow.com/questions/2617835/using-memcached-as-a-session-storage-with-codeigniter 

Having PHP put the sessions into Memcache directly, rather than through framework code is easy - it's just changing two lines in the PHP.ini:

# see http://php.net/manual/en/memcache.ini.php
session.save_handler = memcache
session.save_path="tcp://127.0.0.1:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

This uses the slightly older (but still entirely supported) 'memcache' extension from PECL.

Jenkins delete builds older than latest 20 builds for all jobs

 Ref: https://stackoverflow.com/questions/35610053/jenkins-delete-builds-older-than-latest-20-builds-for-all-jobs

You can use the Jenkins Script Console to iterate through all jobs, get a list of the N most recent and perform some action on the others.

import jenkins.model.Jenkins
import hudson.model.Job

MAX_BUILDS = 20

for (job in Jenkins.instance.items) {
  println job.name

  def recent = job.builds.limit(MAX_BUILDS)

  for (build in job.builds) {
    if (!recent.contains(build)) {
      println "Preparing to delete: " + build
      // build.delete()
    }
  }
}

The Jenkins Script Console is a great tool for administrative maintenance like this and there's often an existing script that does something similar to what you want.

Capacity Planning for MySQL and MariaDB - Dimensioning Storage Size

Ref: https://severalnines.com/database-blog/capacity-planning-mysql-and-mariadb-dimensioning-storage-size

Server manufacturers and cloud providers offer different kinds of storage solutions to cater for your database needs. When buying a new server or choosing a cloud instance to run our database, we often ask ourselves - how much disk space should we allocate? As we will find out, the answer is not trivial as there are a number of aspects to consider. Disk space is something that has to be thought of upfront, because shrinking and expanding disk space can be a risky operation for a disk-based database.

In this blog post, we are going to look into how to initially size your storage space, and then plan for capacity to support the growth of your MySQL or MariaDB database.

How MySQL Utilizes Disk Space

MySQL stores data in files on the hard disk under a specific directory that has the system variable "datadir". The contents of the datadir will depend on the MySQL server version, and the loaded configuration parameters and server variables (e.g., general_log, slow_query_log, binary log).

The actual storage and retrieval information is dependent on the storage engines. For the MyISAM engine, a table's indexes are stored in the .MYI file, in the data directory, along with the .MYD and .frm files for the table. For InnoDB engine, the indexes are stored in the tablespace, along with the table. If innodb_file_per_table option is set, the indexes will be in the table's .ibd file along with the .frm file. For the memory engine, the data are stored in the memory (heap) while the structure is stored in the .frm file on disk. In the upcoming MySQL 8.0, the metadata files (.frm, .par, dp.opt) are removed with the introduction of the new data dictionary schema.

It's important to note that if you are using InnoDB shared tablespace for storing table data (innodb_file_per_table=OFF), your MySQL physical data size is expected to grow continuously even after you truncate or delete huge rows of data. The only way to reclaim the free space in this configuration is to export, delete the current databases and re-import them back via mysqldump. Thus, it's important to set innodb_file_per_table=ON if you are concerned about the disk space, so when truncating a table, the space can be reclaimed. Also, with this configuration, a huge DELETE operation won't free up the disk space unless OPTIMIZE TABLE is executed afterward.

MySQL stores each database in its own directory under the "datadir" path. In addition, log files and other related MySQL files like socket and PID files, by default, will be created under datadir as well. For performance and reliability reason, it is recommended to store MySQL log files on a separate disk or partition - especially the MySQL error log and binary logs.

Database Size Estimation

The basic way of estimating size is to find the growth ratio between two different points in time, and then multiply that with the current database size. Measuring your peak-hours database traffic for this purpose is not the best practice, and does not represent your database usage as a whole. Think about a batch operation or a stored procedure that runs at midnight, or once a week. Your database could potentially grow significantly in the morning, before possibly being shrunk by a housekeeping operation at midnight.

One possible way is to use our backups as the base element for this measurement. Physical backup like Percona Xtrabackup, MariaDB Backup and filesystem snapshot would produce a more accurate representation of your database size as compared to logical backup, since it contains the binary copy of the database and indexes. Logical backup like mysqldump only stores SQL statements that can be executed to reproduce the original database object definitions and table data. Nevertheless, you can still come out with a good growth ratio by comparing mysqldump backups.

We can use the following formula to estimate the database size:

Where,

• Bⁿ - Current week full backup size,
• B^n-1 - Previous week full backup size,
• Db_data - Total database data size,
• Db_index - Total database index size,
• 52 - Number of weeks in a year,
• Y - Year.

The total database size (data and indexes) in MB can be calculated by using the following statements:

1 2 3 4 5 6

mysql> SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) "DB Size in MB" FROM information_schema.tables; +---------------+ | DB Size in MB | +---------------+ |       2013.41 | +---------------+

The above equation can be modified if you would like to use the monthly backups instead. Change the constant value of 52 to 12 (12 months in a year) and you are good to go.

Also, don't forget to account for innodb_log_file_size x 2, innodb_data_file_path and for Galera Cluster, add gcache.size value.

Binary Logs Size Estimation

Binary logs are generated by the MySQL master for replication and point-in-time recovery purposes. It is a set of log files that contain information about data modifications made on the MySQL server. The size of the binary logs depends on the number of write operations and the binary log format - STATEMENT, ROW or MIXED. Statement-based binary log are usually much smaller as compared to row-based binary log, because it only consists of the write statements while the row-based consists of modified rows information.

The best way to estimate the maximum disk usage of binary logs is to measure the binary log size for a day and multiply it with the expire_logs_days value (default is 0 - no automatic removal). It's important to set expire_logs_days so you can estimate the size correctly. By default, each binary log is capped around 1GB before MySQL rotates the binary log file. We can use a MySQL event to simply flush the binary log for the purpose of this estimation.

Firstly, make sure event_scheduler variable is enabled:

1	mysql> SET GLOBAL event_scheduler = ON;

Then, as a privileged user (with EVENT and RELOAD privileges), create the following event:

1 2 3 4 5

mysql> USE mysql; mysql> CREATE EVENT flush_binlog ON SCHEDULE EVERY 1 HOUR STARTS CURRENT_TIMESTAMP ENDS CURRENT_TIMESTAMP + INTERVAL 2 HOUR COMMENT 'Flush binlogs per hour for the next 2 hours' DO FLUSH BINARY LOGS;

For a write-intensive workload, you probably need to shorten down the interval to 30 minutes or 10 minutes before the binary log reaches 1GB maximum size, then round the output up to an hour. Then verify the status of the event by using the following statement and look at the LAST_EXECUTED column:

1 2 3 4

mysql> SELECT * FROM information_schema.events WHERE event_name='flush_binlog'\G        ...        LAST_EXECUTED: 2018-04-05 13:44:25        ...

Then, take a look at the binary logs we have now:

1 2 3 4 5 6 7 8 9 10 11 12 13

mysql> SHOW BINARY LOGS; +---------------+------------+ | Log_name      | File_size  | +---------------+------------+ | binlog.000001 |        146 | | binlog.000002 | 1073742058 | | binlog.000003 | 1073742302 | | binlog.000004 | 1070551371 | | binlog.000005 | 1070254293 | | binlog.000006 |  562350055 | <- hour #1 | binlog.000007 |  561754360 | <- hour #2 | binlog.000008 |  434015678 | +---------------+------------+

We can then calculate the average of our binary logs growth which is around ~562 MB per hour during peak hours. Multiply this value with 24 hours and the expire_logs_days value:

1 2 3 4 5 6

mysql> SELECT (562 * 24 * @@expire_logs_days); +---------------------------------+ | (562 * 24 * @@expire_logs_days) | +---------------------------------+ |                           94416 | +---------------------------------+

We will get 94416 MB which is around ~95 GB of disk space for our binary logs. Slave's relay logs are basically the same as the master's binary logs, except that they are stored on the slave side. Therefore, this calculation also applies to the slave relay logs.

Spindle Disk or Solid State?

There are two types of I/O operations on MySQL files:

• Sequential I/O-oriented files:
  • InnoDB system tablespace (ibdata)
  • MySQL log files:
    • Binary logs (binlog.xxxx)
    • REDO logs (ib_logfile*)
    • General logs
    • Slow query logs
    • Error log
• Random I/O-oriented files:
  • InnoDB file-per-table data file (*.ibd) with innodb_file_per_table=ON (default).

Consider placing random I/O-oriented files in a high throughput disk subsystem for best performance. This could be flash drive - either SSDs or NVRAM card, or high RPM spindle disks like SAS 15K or 10K, with hardware RAID controller and battery-backed unit. For sequential I/O-oriented files, storing on HDD with battery-backed write-cache should be good enough for MySQL. Take note that performance degradation is likely if the battery is dead.

We will cover this area (estimating disk throughput and file allocation) in a separate post.

Capacity Planning and Dimensioning

Capacity planning can help us build a production database server with enough resources to survive daily operations. We must also provision for unexpected needs, account for future storage and disk throughput needs. Thus, capacity planning is important to ensure the database has enough room to breath until the next hardware refresh cycle.

It's best to illustrate this with an example. Considering the following scenario:

• Next hardware cycle: 3 years
• Current database size: 2013 MB
• Current full backup size (week N): 1177 MB
• Previous full backup size (week N-1): 936 MB
• Delta size: 241MB per week
• Delta ratio: 25.7% increment per week
• Total weeks in 3 years: 156 weeks
• Total database size estimation: ((1177 - 936) x 2013 x 156)/936 = 80856 MB ~ 81 GB after 3 years

If you are using binary logs, sum it up from the value we got in the previous section:

• 81 + 95 = 176 GB of storage for database and binary logs.

Add at least 100% more room for operational and maintenance tasks (local backup, data staging, error log, operating system files, etc):

• 176 + 176 = 352 GB of total disk space.

Based on this estimation, we can conclude that we would need at least 352 GB of disk space for our database for 3 years. You can use this value to justify your new hardware purchase. For example, if you want to buy a new dedicated server, you could opt for 6 x 128 SSD RAID 10 with battery-backed RAID controller which will give you around 384 GB of total disk space. Or, if you prefer cloud, you could get 100GB of block storage with provisioned IOPS for our 81GB database usage and use the standard persistent block storage for our 95GB binary logs and other operational usage.

Happy dimensioning!