osm2pgsql_tuner.tuner

Contains osm2pgsql class to help provide osm2pgsql tuning advice.

Recommendations are targeted for osm2pgsql v1.5.0 and newer.

Module Contents

Classes

Recommendation

Takes basic inputs to generate command recommendations for osm2pgsql.

Attributes

FLAT_NODES_THRESHOLD_GB

Sets threshold size for when to use --flat-nodes.
osm2pgsql_tuner.tuner.FLAT_NODES_THRESHOLD_GB = 8.0[source]

Sets threshold size for when to use --flat-nodes.

8-10 GB appears to be an appropriate threshold when using modern SSDs for storage.

Type:

float

class osm2pgsql_tuner.tuner.Recommendation(system_ram_gb: float, osm_pbf_gb: float, slim_no_drop: bool = False, append_first_run: bool = None, pgosm_layer_set: str = 'run', ssd: bool = True)[source]

Takes basic inputs to generate command recommendations for osm2pgsql.

Parameters:

  • system_ram_gb (float) – How much total RAM the server has, in GB.

  • osm_pbf_gb (float) – Size of the .osm.pbf file in GB.

  • slim_no_drop (bool) – (Default False) Setup to use osm2pgsql --append. When true --slim must be used.

  • append_first_run (bool or None) – (Default None) If slim_no_drop is True, append_first_run must be set. Used to determine if --create or --append is passed to osm2pgsql.

  • pgosm_layer_set (str) –

    (Default (run) Base name of .lua script to run.

    PgOSM Flex uses the run entry point.

  • ssd (bool) – (Default True) Is the osm2pgsql server using SSD for storage? Value determines threshold for decision to use --flat-nodes

limited_ram_check() → bool[source]

Decide if osm2pgsql can use more RAM than the system has available.

Returns:

limited_ram

Return type:

boolean

use_flat_nodes() → bool[source]

Returns True if --flat-nodes should be used.

Use --flat-nodes when:

  • PBF size is larger than config’d threshold AND SSD

  • PBF >= 30 GB (regardless of SSD)

If the load can run entirely in-memory, no need to use flat nodes.

Returns:

use_flat_nodes

Return type:

bool

use_drop() → bool[source]

Checks other parameters to determine if --drop should be used.

Returns:

use_drop

Return type:

bool

calculate_max_osm2pgsql_cache() → float[source]

Calculates the max RAM server has available to dedicate to osm2pgsql cache.

Using 2/3 of reported system total.

osm2pgsql_cache_max : float

get_osm2pgsql_noslim_cache() → float[source]

Calculates cache required by osm2pgsql in order to run w/out slim.

Uses basic calculation based on the size of the PBF size being imported.

Justification: https://blog.rustprooflabs.com/2021/05/osm2pgsql-reduced-ram-load-to-postgis

Returns:

required_gb – Estimated memory (in GB) osm2pgsql will use if running w/out slim mode.

Return type:

float

run_in_ram() → bool[source]

Determines if bypassing --slim is an option with the given details.

Uses details about append mode, RAM available and the size of the input PBF to make determination.

Returns:

in_ram_possible

Return type:

bool

get_osm2pgsql_command(pbf_path: str) → str[source]

Builds the recommended osm2pgsql command.

Parameters:

pbf_path (str) –

Returns:

cmd

Return type:

str

get_cache_mb() → int[source]

Returns cache size to set in MB.

osm2pgsql will only use a cache value > 0 while in slim mode.

Returns:

cache – Size in MB to set –cache

Return type:

int