Hosting a shared MySQL instance for your internal or external clients (“multi-tenant”) was always a challenge. Multi-tenants approach or a “schema-per-customer” approach is pretty common nowadays to host multiple clients on the same MySQL sever. One of issues of this approach, however, is the lack of visibility: it is hard to tell how many resources (queries, disk, cpu, etc) each user will use.
Percona Server contains userstats Google patch, which will allow you to get the resource utilization per user. The new MySQL 5.6 performance_schema has even more instrumentation which can give you a better visibility on per-user or per-schema/per-database level. And if you are running MySQL 5.6.6 or higher, Performance Schema is already enabled (with minimum set of “instrumentation” thou) and ready to use. In this post I will share examples of using Performance Schema for multi-tenant environments.
Overview
If you want to use Performance Schema, make sure it is enabled (enabled by default starting with MySQL 5.6.6):
mysql> show global variables like 'performance_schema'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | performance_schema | ON | +--------------------+-------+ 1 row in set (0.00 sec)
Performance_schema provides you with the “raw” metrics and it may be difficult to select data from it. The good news is that you can use the “sys” schema project by Mark Leith. The “sys” schema (old name: ps_helper) is a collection of views and stored procedures which will provide you with reports in human readable format. The installation is easy, download it from github and run:
$ mysql -u root -p < ./sys_56.sql
(it will only create database “sys” and a set of views and stored procedures/stored functions in it)
Examples
For the multi-tenant environment the most interesting is resource utilization breakdown. Lets say you want to “charge per IO” similar to Amazon RDS for MySQL model. You can now run this simple query against “sys” schema, which will use Performance Schema in MySQL 5.6 to generate report (in my example all users starts with “a”):
mysql sys> select * from user_summary_by_file_io where user like 'a%'; +-------+------------+-------------+ | user | ios | io_latency | +-------+------------+-------------+ | api01 | 3061469563 | 4.04h | | api03 | 912296937 | 1.36h | | api02 | 815473183 | 1.22h | | app01 | 9704381 | 00:01:06.33 | | app02 | 1160149 | 8.18 s | | app03 | 1424065 | 7.23 s | +-------+------------+-------------+ 6 rows in set (0.06 sec)
If you need more extensive metrics you can use this report:
mysql sys> select * from user_summary where user like 'a%' limit 10; +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ | user | statements | statement_latency | statement_avg_latency | table_scans | file_ios | file_io_latency | current_connections | total_connections | unique_hosts | +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ | api01 | 1837898366 | 192.29h | 376.65 us | 2221018 | 3064724488 | 4.05h | 22 | 73748519 | 1 | | api02 | 833712609 | 206.83h | 893.11 us | 1164 | 836753563 | 1.26h | 7 | 246730 | 1 | | api03 | 988875004 | 222.03h | 808.29 us | 1156 | 943363030 | 1.43h | 8 | 247639 | 1 | | app01 | 1864159 | 4.01h | 7.75 ms | 46756 | 9704623 | 00:01:06.33 | 0 | 300004 | 1 | | app02 | 812520 | 00:20:51.69 | 1.54 ms | 42513 | 1164954 | 8.37 s | 0 | 122994 | 1 | | app03 | 811558 | 00:21:59.57 | 1.63 ms | 42310 | 1429373 | 7.40 s | 0 | 123110 | 1 | +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ 6 rows in set
Or a breakdown per user per statement:
mysql sys> select user, statement, total, rows_sent, rows_examined, rows_affected, full_scans from user_summary_by_statement_type where user = 'api01' limit 100; +-------+-------------+-----------+-----------+---------------+---------------+------------+ | user | statement | total | rows_sent | rows_examined | rows_affected | full_scans | +-------+-------------+-----------+-----------+---------------+---------------+------------+ | api01 | insert | 368405774 | 0 | 0 | 494179199 | 0 | | api01 | commit | 376276686 | 0 | 0 | 0 | 0 | | api01 | select | 476129532 | 761038004 | 1067363348 | 0 | 0 | | api01 | update | 142896430 | 0 | 142896437 | 141978066 | 0 | | api01 | begin | 376280424 | 0 | 0 | 0 | 0 | | api01 | show_fields | 2215016 | 310102240 | 310102240 | 0 | 2215016 | | api01 | Quit | 73421361 | 0 | 0 | 0 | 0 | | api01 | change_db | 17258767 | 0 | 0 | 0 | 0 | | api01 | delete | 7896 | 0 | 7896 | 7896 | 0 | | api01 | rollback | 3711 | 0 | 0 | 0 | 0 | +-------+-------------+-----------+-----------+---------------+---------------+------------+ 10 rows in set (0.04 sec)
If you are using “schema per customer” approach you get get the per-schema resource utilization using sys.schema_table_statistics. Example (from Peter Zaitsev’s webinar):
mysql [localhost] {msandbox} > select * from schema_table_statistics where table_schema='sbtest' limit 1 G
*************************** 1. row ***************************
table_schema: sbtest
table_name: sbtest
rows_fetched: 158764154
fetch_latency: 1.37h
rows_inserted: 378901
insert_latency: 00:07:17.38
rows_updated: 1136714
update_latency: 00:45:40.08
rows_deleted: 378902
delete_latency: 00:03:00.34
io_read_requests: 636003
io_read: 9.70 GiB
io_read_latency: 00:28:12.01
io_write_requests: 203925
io_write: 3.11 GiB
io_write_latency: 17.26 s
io_misc_requests: 2449
io_misc_latency: 3.87 sThis report may be really slow if you have lots of tables. If you are only interested in disk utilization per database you can directly query the performance_schema:
SELECT extract_schema_from_file_name(fsbi.file_name) AS table_schema,
SUM(fsbi.count_read) AS io_read_requests,
sys.format_bytes(SUM(fsbi.sum_number_of_bytes_read)) AS io_read,
sys.format_time(SUM(fsbi.sum_timer_read)) AS io_read_latency,
SUM(fsbi.count_write) AS io_write_requests,
sys.format_bytes(SUM(fsbi.sum_number_of_bytes_write)) AS io_write,
sys.format_time(SUM(fsbi.sum_timer_write)) AS io_write_latency,
SUM(fsbi.count_misc) AS io_misc_requests,
sys.format_time(SUM(fsbi.sum_timer_misc)) AS io_misc_latency
FROM performance_schema.file_summary_by_instance AS fsbi
GROUP BY table_schema
ORDER BY io_write_requests DESC limit 10;
+--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+
| table_schema | io_read_requests | io_read | io_read_latency | io_write_requests | io_write | io_write_latency | io_misc_requests | io_misc_latency |
+--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+
| binlogs | 6943921992 | 19.57 TiB | 2.46h | 1124408959 | 6.60 TiB | 2.18h | 914411 | 00:03:55.75 |
| d1 | 146448 | 2.24 GiB | 1.97 s | 22768241 | 241.44 GiB | 00:12:57.97 | 5095817 | 00:12:50.16 |
| stats1 | 853 | 1.81 MiB | 97.12 ms | 1042768 | 15.93 GiB | 10.27 s | 292303 | 22.74 s |
| stats2 | 3024 | 11.22 MiB | 781.96 ms | 750182 | 11.45 GiB | 9.32 s | 267780 | 50.84 s |
| relaylogs | 105065 | 925.71 MiB | 312.82 ms | 447607 | 925.71 MiB | 1.19 s | 72169 | 530.70 ms |
| o1268 | 2078 | 3.02 MiB | 7.40 ms | 1591 | 203.97 KiB | 1.17 ms | 2693 | 92.34 ms |
+--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+Edit: if you also need to get a quick “hot tables” report with focus on disk utilization (io requests) you can use this query:
SELECT concat(extract_schema_from_file_name(fsbi.file_name), '.',
extract_table_from_file_name(fsbi.file_name) ) as unique_table_name,
SUM(fsbi.count_read) AS io_read_requests,
sys.format_bytes(SUM(fsbi.sum_number_of_bytes_read)) AS io_read,
sys.format_time(SUM(fsbi.sum_timer_read)) AS io_read_latency,
SUM(fsbi.count_write) AS io_write_requests,
sys.format_bytes(SUM(fsbi.sum_number_of_bytes_write)) AS io_write,
sys.format_time(SUM(fsbi.sum_timer_write)) AS io_write_latency,
SUM(fsbi.count_misc) AS io_misc_requests,
sys.format_time(SUM(fsbi.sum_timer_misc)) AS io_misc_latency
FROM performance_schema.file_summary_by_instance AS fsbi
GROUP BY unique_table_name
ORDER BY io_write_requests DESC limit 10;This will give you a breakdown per table (and will include system files as well).
Conclusion
Performance_schema in MySQL 5.6 is very useful tool and can help you to get better visualization in MySQL multi-user (multi-tenant) installations. It can also do much more, i.e. find all queries with temporary tables or profile locks and mutex. “sys” schema provide you with the useful pre-defined reports, the table (view) names are almost self explanatory:
+-----------------------------------------------+ | Tables_in_sys | +-----------------------------------------------+ | innodb_buffer_stats_by_schema | | innodb_buffer_stats_by_table | ... | processlist | ... | schema_tables_with_full_table_scans | | schema_unused_indexes | | statement_analysis | | statements_with_errors_or_warnings | | statements_with_full_table_scans | | statements_with_runtimes_in_95th_percentile | | statements_with_sorting | | statements_with_temp_tables | ...
If you are using MySQL 5.6, performance_schema will give a additional valuable inside.
The post Using MySQL 5.6 Performance Schema in multi-tenant environments appeared first on MySQL Performance Blog.