Support ingesting SST files generated by a live DB (#12750)

Summary:
... to enable use cases like using RocksDB to merge sort data for ingestion. A new file ingestion option `IngestExternalFileOptions::allow_db_generated_files` is introduced to allows users to ingest SST files generated by live DBs instead of SstFileWriter. For now this only works if the SST files being ingested have zero as their largest sequence number AND do not overlap with any data in the DB (so we can assign seqno 0 which matches the seqno of all ingested keys).

The feature is marked the option as experimental for now.

Main changes needed to enable this:
- ignore CF id mismatch during ingestion
- ignore the missing external file version table property

Rest of the change is mostly in new unit tests.

A previous attempt is in https://github.com/facebook/rocksdb/issues/5602.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12750

Test Plan: - new unit tests

Reviewed By: ajkr, jowlyzhang

Differential Revision: D58396673

Pulled By: cbi42

fbshipit-source-id: aae513afad7b1ff5d4faa48104df5f384926bf03

1 files changed, 18 insertions, 0 deletions

diff --git a/include/rocksdb/options.h b/include/rocksdb/options.h
index 6a3e5461d..0018d9f80 100644
--- a/include/rocksdb/options.h
+++ b/include/rocksdb/options.h
@@ -2179,6 +2179,24 @@ struct IngestExternalFileOptions {
   //
   // XXX: "bottommost" is obsolete/confusing terminology to refer to last level
   bool fail_if_not_bottommost_level = false;
+  // EXPERIMENTAL
+  // If set to true, ingestion will
+  // - allow the files to not be generated by SstFileWriter, and
+  // - ignore cf_id mismatch between cf_id in the files and the CF they are
+  // being ingested into.
+  //
+  // REQUIRES:
+  // - files to be ingested do not overlap with existing keys.
+  // - write_global_seqno = false
+  // - move_files = false
+  //
+  // Warning: This ONLY works for SST files where all keys have sequence number
+  // zero and with no duplicated user keys (this should be guaranteed if the
+  // file is generated by a DB with zero as the largest sequence number).
+  // We scan the entire SST files to validate sequence numbers.
+  // Warning: If a DB contains ingested files generated by another DB/CF,
+  // RepairDB() may not correctly recover these files. It may lose these files.
+  bool allow_db_generated_files = false;
 };
 
 enum TraceFilterType : uint64_t {