{"id":540,"date":"2022-10-09T10:03:14","date_gmt":"2022-10-09T10:03:14","guid":{"rendered":"https:\/\/blog.get-map.org\/?p=540"},"modified":"2022-10-09T10:03:14","modified_gmt":"2022-10-09T10:03:14","slug":"evil-hack-of-the-week-speeding-up-locale-gen","status":"publish","type":"post","link":"https:\/\/blog.get-map.org\/index.php\/2022\/10\/09\/evil-hack-of-the-week-speeding-up-locale-gen\/","title":{"rendered":"Evil hack of the week: Speeding up locale-gen"},"content":{"rendered":"<p><a name=\"preamble\"><\/a> <\/p>\n<p>The <code>locale-gen<\/code> script can take some time, especially when trying to generate <strong>all<\/strong> supported locales.<\/p>\n<p>Part of the problem is that it only runs the actual <code>localedef<\/code> tool for one locale at a time, so utilizing only one CPU core at a time.<\/p>\n<p><!--more--><\/p>\n<p>So what can be done to improve the situation, preferably without touching the distributions script itself?<\/p>\n<p>This is what I ended up doing when provisioning virtual machines:<\/p>\n<table border=\"0\" bgcolor=\"#e8e8e8\" width=\"100%\" style=\"margin:0.2em 0;\">\n<tr>\n<td style=\"padding:0.5em;\">\n<pre style=\"margin:0; padding:0;\"># create a unique temporary directory based on current PID\nmkdir \/tmp\/locales.$$\n\n# now do everything else in a sub shell to avoid changing\n# the current shells environment\n(\n  cd \/tmp\/locales.$$\n\n  # create a local \"localedef\" wrapper script that justs\n  # prints the actual command to be run\n  echo 'echo; echo \/usr\/bin\/localedef \"$@\"' &gt; localedef\n  chmod a+x localedef\n\n  # make sure our wrapper script is found first\n  export PATH=.:$PATH\n\n  # just make sure that we are not going to get warnings\n  # about \"locale not found\" while running \"locale-gen\"\n  # the \"C\" locale should always be there\n  export LANG=C\n  export LC_ALL=C\n\n  # now run \"locale-gen\", filter out the \"localedef\"\n  # commands only (there is no way to otherwise silence\n  # local-gen progress messages) and feed the printed\n  # commands into GNU parallel to utilize all CPU cores\n  locale-gen | egrep \"^\/usr\" | parallel\n)\n\n# cleanup\nrm -rf \/tmp\/locales.$$<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<p>The prerequisite to this is obviously to have the GNU parallel tool installed.<\/p>\n<p>With that in place the wall clock time needed to (re-)generate all supported locales is drastically reduced, and all CPU cores can be seen working close to 100% while running this instead of just one.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The locale-gen script can take some time, especially when trying to generate all supported locales. Part of the problem is that it only runs the actual localedef tool for one locale at a time, so utilizing only one CPU core at a time.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/posts\/540"}],"collection":[{"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/comments?post=540"}],"version-history":[{"count":1,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/posts\/540\/revisions"}],"predecessor-version":[{"id":541,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/posts\/540\/revisions\/541"}],"wp:attachment":[{"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/media?parent=540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/categories?post=540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.get-map.org\/index.php\/wp-json\/wp\/v2\/tags?post=540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}