Importgeschwindigkeit von XT

[Saunders Computer]

Hallo miteinander,

ich habe einen XTC-Shop 3.04 SP 2.1 auf einem root-Server laufen (Intel i 920 4x2,5 Ghz, 8 GB Ram, 750 GB Raid) und habe Probleme mit der Importgeschwindigkeit.

Beim Neueinspielen von Artikeln benötigt ein 1000 Artikel Paket ca. 30-60 Sekunden, ein Update von bestehenden Artikeln nimmt pro 1000 Artikel bis zu 5 Minuten in Anspruch. Insgesamt sind derzeit ca. 60000 Artikel im Shop.
So dauert die Update-Prozedur täglich fast zwei Stunden.

Bei einem Gambio System, das auf einem stinknormalen shared-hosting System installiert ist, ist der Import von rund 17.000 Artikeln, egal ob Update oder Neueinspielung, mühelos innerhalb von 10 Minuten durch. Das 1000 Artikelpaket braucht nur wenige Sekunden.

Da der Artikelbestand noch erweitert werden soll, bin ich zunehmend dazu gezwungen, den Import zu optimieren. Mich wundert es das das Gambio System so schnell arbeitet im Gegensatz zu dem XT-Shop auf dem root-Server.

Ich wäre sehr dankbar, wenn jemand einen Tipp für mich hätte!

mfg
Saunders

[Fry]

Ich gehe mal davon aus, dass das Webspacepaket ganz einfach vom Provider gepflegt wird, der Server aber nur eine Standartinstallation vom OS und evtl. nachträglich installierten Patches und/oder sonstiger Software hat. Kein apache, kein SQL optimiert, evtl. auch noch unnötiger Ballast in Form von unnötigen und ressourcenfressenden Modulen, was einen Flaschenhals hat, dann dauert der Import dementsprechend.

Evtl. die ganzen Konfigurationen durchsehen mit tuning-primer z.B?!

Wäre meine erste Richtung, wonach ich suchen täte, da ich das immer öfter sehe bei Leuten mit "eigenen" Servern, Standart-OS und dann klatsch bumm.

[Saunders Computer]

Danke für den Tipp... tuning primer

Könnt Ihr mir sagen wie lange es bei euch dauert, für ein paar Vergleichswerte damit ich weiß wo ich stehe.

Mir liegt nahe denselben Gambio-Shop mal auf dem Server auszuprobieren. Die Leistung sollte doch ausreichen oder... hat der Webhoster einen 10 mal schnelleren? Man hat aber dort nicht viel Arbeitsspeicher zur Verfügung.

Gibt es irgendwelche Shop-Import Module zum nachinstallieren, die besser sind als das bereitgestellte?

Welche SQL Einstellungen kommen denn am ehesten in Frage?

Vielen Dank

[Saunders Computer]

tuning-primer liefert folgendes Ergebnis:

MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -

MySQL Version 5.0.51a-24+lenny1 x86_64

Uptime = days hrs min sec
Avg. qps = 21
Total Questions = 87849261
Threads Connected = 2

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
MySQL :: MySQL 5.0 Reference Manual :: 5.1.3 Server System Variables
Visit MySQL :: MySQL Enterprise Advisors
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10 sec.
You have 0 out of 87849282 that take longer than 10 sec. to complete
Your long_query_time may be too high, I typically set this under 5 sec.

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See MySQL :: MySQL 5.0 Reference Manual :: 6.3 Point-in-Time Recovery

WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 7
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 100
Current threads_connected = 2
Historic max_used_connections = 22
The number of used connections is 22% of the configured maximum.
Your max_connections variable seems to be fine.

MEMORY USAGE
Max Memory Ever Allocated : bytes
Configured Max Per-thread Buffers : bytes
Configured Max Global Buffers : bytes
Configured Max Memory Limit : bytes
Physical Memory : G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = M
Current key_buffer_size = M
Key cache miss rate is 1 : 86
Key buffer free ratio = %
Your key_buffer_size seems to be fine

QUERY CACHE
Query cache is enabled
Current query_cache_size = M
Current query_cache_used = M
Current query_cache_limit = M
Current Query cache Memory fill ratio = %
Current query_cache_min_res_unit = K
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = M
Current read_rnd_buffer_size = K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = K
You have had 723205 queries where a join could not use an index properly
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

Note! This script will still suggest raising the join_buffer_size when
ANY joins not using indexes are found.

OPEN FILES LIMIT
Current open_files_limit = 1024 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_cache value = 64 tables
You have a total of 289 tables
You have 64 open tables.
Current table_cache hit rate is 0%, while 100% of your table cache is in use
You should probably increase your table_cache

TEMP TABLES
Current max_heap_table_size = M
Current tmp_table_size = M
Of 1986811 temp tables, 19% were created on disk
Effective in-memory tmp_table_size is limited to max_heap_table_size.
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = K
Current table scan ratio = 37968 : 1
You have a high ratio of sequential access requests to SELECTs
You may benefit from raising read_buffer_size and/or improving your use of indexes.

TABLE LOCKING
Current Lock Wait ratio = 1 : 467
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=2'.

[Saunders Computer]

Und hier sind die Variablenwerte von mysqladmin:

Brauche bitte hier euren Rat, um mit guten SQL Einstellungen den Server voll (!) auszulasten.
Es läuft nur der Shop alleine auf dem Server.

+---------------------------------+-----------------------------+
| Variable_name | Value |
+---------------------------------+-----------------------------+
| auto_increment_increment | 1 |
| auto_increment_offset | 1 |
| automatic_sp_privileges | ON |
| back_log | 50 |
| basedir | /usr/ |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 5 |
| datadir | /var/lib/mysql/ |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| div_precision_increment | 4 |
| keep_files_on_create | OFF |
| engine_condition_pushdown | OFF |
| expire_logs_days | 10 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
| group_concat_max_len | 1024 |
| have_archive | YES |
| have_bdb | NO |
| have_blackhole_engine | YES |
| have_compress | YES |
| have_crypt | YES |
| have_csv | YES |
| have_dynamic_loading | YES |
| have_example_engine | NO |
| have_federated_engine | YES |
| have_geometry | YES |
| have_innodb | YES |
| have_isam | NO |
| have_merge_engine | YES |
| have_ndbcluster | DISABLED |
| have_openssl | DISABLED |
| have_ssl | DISABLED |
| have_query_cache | YES |
| have_raid | NO |
| have_rtree_keys | YES |
| have_symlink | YES |
| hostname | 188.40.70.203 |
| init_connect | |
| init_file | |
| init_slave | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 8388608 |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_io_threads | 4 |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_open_files | 300 |
| innodb_rollback_on_timeout | OFF |
| innodb_support_xa | ON |
| innodb_sync_spin_loops | 20 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| innodb_thread_sleep_delay | 10000 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 16777216 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| language | /usr/share/mysql/english/ |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| lc_time_names | en_US |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_error | |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 16776192 |
| max_binlog_cache_size | 18446744073709551615 |
| max_binlog_size | 104857600 |
| max_connect_errors | 10 |
| max_connections | 100 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 18446744073709551615 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 18446744073709551615 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 18446744073709551615 |
| multi_range_count | 256 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 9223372036854775807 |
| myisam_recover_options | BACKUP |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 8388608 |
| myisam_stats_method | nulls_unequal |
| ndb_autoincrement_prefetch_sz | 32 |
| ndb_force_send | ON |
| ndb_use_exact_count | ON |
| ndb_use_transactions | ON |
| ndb_cache_check_time | 0 |
| ndb_connectstring | |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | OFF |
| open_files_limit | 1024 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| pid_file | /var/run/mysqld/mysqld.pid |
| port | 3306 |
| preload_buffer_size | 32768 |
| profiling | OFF |
| profiling_history_size | 15 |
| protocol_version | 10 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 16777216 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 131072 |
| read_only | OFF |
| read_rnd_buffer_size | 262144 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| secure_file_priv | |
| server_id | 0 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_compressed_protocol | OFF |
| slave_load_tmpdir | /tmp/ |
| slave_net_timeout | 3600 |
| slave_skip_errors | OFF |
| slave_transaction_retries | 10 |
| slow_launch_time | 2 |
| socket | /var/run/mysqld/mysqld.sock |
| sort_buffer_size | 2097144 |
| sql_big_selects | ON |
| sql_mode | |
| sql_notes | ON |
| sql_warnings | OFF |
| ssl_ca | |
| ssl_capath | |
| ssl_cert | |
| ssl_cipher | |
| ssl_key | |
| storage_engine | MyISAM |
| sync_binlog | 0 |
| sync_frm | ON |
| system_time_zone | CEST |
| table_cache | 64 |
| table_lock_wait_timeout | 50 |
| table_type | MyISAM |
| thread_cache_size | 8 |
| thread_stack | 131072 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| timed_mutexes | OFF |
| tmp_table_size | 33554432 |
| tmpdir | /tmp |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| updatable_views_with_limit | YES |
| version | 5.0.51a-24+lenny1 |
| version_comment | (Debian) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
| wait_timeout | 28800 |
+---------------------------------+-----------------------------+

[Saunders Computer]

ich habe nun einige SQL Optimierungen vorgenommen...

Eine Testinstallation von Gambio auf dem Server brachte das Ergebnis:
50000 Artikel werden in unter drei Minuten importiert und geupdated.
Während XT-Commerce sich weiterhin pro 1000 Artikel bis zu 5 Minuten gönnt.

Eigentlich will ich aber mit XT weiterarbeiten. Bitte helft mir.

[Fry]

Um zu verstehen wie genau SQL und der Indianer was wie wo eingestellt haben wollen, einfach mal hier vorbeischauen: MySQL Performance Tuning mit Tuning-Primer.sh Script - Server Support Forum

Und wenn du schon mal dort bist, auch die Seite von Huschi mal anschauen, dort sind auch einige Tipps zu finden (ich bin nicht Huschi!).

Bis bei mir alles richtig lief, hab ich fast vier Wochen gebraucht, hab mich aber dank des SSF recht gut damit auseinandergesetzt.