(embulk-input-jdbc) -

[sureshkmr761]

Hello, if we use use_raw_query_with_incremental: option, we have to supply last_record option. As this option would stay static in the config file. So the first execution would use this value but next scheduled executions will start fetching the data from this value or the actual last transaction and ignore this option or overwrite for subsequent runs?

[hiroyuki-sato]

Hello, @sureshkmr761.

If you use use_raw_query_with_incremental, you need to manage the last_record value by yourself (Outside of Embulk command).

Executing embulk run config -c last_record.yml merges two configuration data and execute the task.

You just update last_record.yaml using Workflow tools.

An alternative option use liquid template engine. (Please see, Using variables section)

config.yml

in:
  type: postgresql
  host: host
  password: password
  user: user
  database: embulk_test
  use_raw_query_with_incremental: true
  incremental: true
  incremental_columns:
  - bar_id
  query: |
   select
     id as bar_id,
     col_text,
     col_double,
     col_time
    from sample where id > :bar_id
  #last_record: [1234]

out: {type: stdout}

last_record.yaml

in:
  last_record: [4567]

[dmikurube]

Let us move this type of questions to User forum.

Thanks for your follow-up, @hiroyuki-sato.

[dmikurube]

We see no response for a year. Let me close this discussion.