Documentation of 'org.apache.derby.impl.store.raw.log.LogToFile' Java class
LogToFile
Skip navigation links
org.apache.derby.impl.store.raw.log

Class LogToFile

  • All Implemented Interfaces:
    java.security.PrivilegedExceptionAction<java.lang.Object>, Serviceable, ModuleControl, ModuleSupportable, Corruptable, LogFactory

    public final class LogToFile
extends java.lang.Object
implements LogFactory, ModuleControl, ModuleSupportable, Serviceable, java.security.PrivilegedExceptionAction<java.lang.Object>
    This is an implementation of the log using a non-circular file system file. No support for incremental log backup or media recovery. Only crash recovery is supported.

    The 'log' is a stream of log records. The 'log' is implemented as a series of numbered log files. These numbered log files are logically continuous so a transaction can have log records that span multiple log files. A single log record cannot span more then one log file. The log file number is monotonically increasing.

    The log belongs to a log factory of a RawStore. In the current implementation, each RawStore only has one log factory, so each RawStore only has one log (which composed of multiple log files). At any given time, a log factory only writes new log records to one log file, this log file is called the 'current log file'.

    A log file is named loglogNumber.dat

    Everytime a checkpoint is taken, a new log file is created and all subsequent log records will go to the new log file. After a checkpoint is taken, old and useless log files will be deleted.

    RawStore exposes a checkpoint method which clients can call, or a checkpoint is taken automatically by the RawStore when

    1. the log file grows beyond a certain size (configurable, default 100K bytes)
    2. RawStore is shutdown and a checkpoint hasn't been done "for a while"
    3. RawStore is recovered and a checkpoint hasn't been done "for a while"

    This LogFactory is responsible for the formats of 2 kinds of file: the log file and the log control file. And it is responsible for the format of the log record wrapper. 

    
        Format of log control file

    • Field Detail

      • DBG_FLAG

        public static final java.lang.String DBG_FLAG

      • DUMP_LOG_ONLY

        public static final java.lang.String DUMP_LOG_ONLY

      • DUMP_LOG_FROM_LOG_FILE

        public static final java.lang.String DUMP_LOG_FROM_LOG_FILE

      • TEST_LOG_SWITCH_LOG

        public static final java.lang.String TEST_LOG_SWITCH_LOG
        Set to true if we want the checkpoint to only switch the log but not actually do the checkpoint

      • TEST_LOG_INCOMPLETE_LOG_WRITE

        public static final java.lang.String TEST_LOG_INCOMPLETE_LOG_WRITE
        Set to true if we want the upcoming log record to be only partially written. The database is corrupted if not immediately shutdown. Set TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES to the number of bytes to write out, default is 1 byte.

      • TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES

        public static final java.lang.String TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES
        Set to the number of bytes we want the next log record to actually write out, only used when TEST_LOG_INCOMPLETE_LOG_WRITE is on. Default is 1 byte.

      • TEST_LOG_FULL

        public static final java.lang.String TEST_LOG_FULL
        Set to true if we want to simulate a log full condition

      • TEST_SWITCH_LOG_FAIL1

        public static final java.lang.String TEST_SWITCH_LOG_FAIL1
        Set to true if we want to simulate a log full condition while switching log

      • TEST_SWITCH_LOG_FAIL2

        public static final java.lang.String TEST_SWITCH_LOG_FAIL2

      • TEST_RECORD_TO_FILL_LOG

        public static final java.lang.String TEST_RECORD_TO_FILL_LOG
        Set to the number of log record we want to write before the log is simulated to be full.

      • TEST_MAX_LOGFILE_NUMBER

        public static final java.lang.String TEST_MAX_LOGFILE_NUMBER
        Set to true if we want to simulate max possible log file number is being used.

    • Constructor Detail

      • LogToFile

        public LogToFile()
        MT- not needed for constructor

    • Method Detail

      • getTypeFormatId

        public int getTypeFormatId()
        Return my format identifier.

      • checkpoint

        public boolean checkpoint(RawStoreFactory rsf,
                          DataFactory df,
                          TransactionFactory tf,
                          boolean wait)
                   throws StandardException
        Checkpoint the rawStore.

        MT- Only one checkpoint is to be taking place at any given time.

        The steps of a checkpoint are

        1. switch to a new log file if possible 
                                  freeze the log (for the transition to a new log file)
                                flush current log file
                                create and flush the new log file (with file number 1 higher
                than the previous log file). The new log file becomes the
                current log file.
                        unfreeze the log
        2. start checkpoint transaction
        3. gather interesting information about the rawStore: the current log instant (redoLWM) the earliest active transaction begin tran log record instant (undoLWM), all the truncation LWM set by clients of raw store (replication)
        4. clean the buffer cache
        5. log the next checkpoint log record, which contains (repPoint, undoLWM, redoLWM) and commit checkpoint transaction.
        6. synchronously write the control file containing the next checkpoint log record log instant
        7. the new checkpoint becomes the current checkpoint. Somewhere near the beginning of each log file should be a checkpoint log record (not guarenteed to be there)
        8. see if the log can be truncated

          The earliest useful log record is determined by the repPoint and the undoLWM, whichever is earlier.

          Every log file whose log file number is smaller than the earliest useful log record's log file number can be deleted. 

                                  Transactions can be at the following states w/r to a checkpoint -
                        consider the log as a continous stream and not as series of log 
            files for the sake of clarity.  
                        |(BT)-------(ET)| marks the begin and end of a transaction.
                        .                          checkpoint started
                        .       |__undoLWM          |
                        .       V                   |___redoLWM
                        .                           |___TruncationLWM
                        .                           |
                        .                           V
                        1 |-----------------|
                        2       |--------------------------------|
                        3           |-------|
                        4               |--------------------------------------(end of log)
                        5                                       |-^-|
                        .                                   Checkpoint Log Record
                        ---A--->|<-------B--------->|<-------------C-----------

          There are only 3 periods of interest :
          A) before undoLWM, B) between undo and redo LWM, C) after redoLWM.

          Transaction 1 started in A and terminates in B.
          During redo, we should only see log records and endXact from this transaction in the first phase (between undoLWM and redoLWM). No beginXact log record for this transaction will be seen.

          Transaction 2 started in B (right on the undoLWM) and terminated in C.
          Any transaction that terminates in C must have a beginXact at or after undoLWM. In other words, no transaction can span A, B and C. During redo, we will see beginXact, other log records and endXact for this transaction.

          Transaction 3 started in B and ended in B.
          During redo, we will see beginXact, other log records and endXact for this transaction.

          Transaction 4 begins in B and never ends.
          During redo, we will see beginXact, other log records. In undo, this loser transaction will be rolled back.

          Transaction 5 is the transaction taking the checkpoint.
          The checkpoint action started way back in time but the checkpoint log record is only written after the buffer cache has been flushed.

          Note that if any time elapse between taking the undoLWM and the redoLWM, then it will create a 4th period of interest.

        Specified by:
        checkpoint in interface LogFactory
        Parameters:
        rsf - The RawStoreFactory to use to do the checkpoint.
        df - The DataFactory to use to do the checkpoint.
        tf - The TransactionFactory to use to do the checkpoint.
        wait - If an existing checkpoint is in progress, then if wait=true then this routine will wait for the checkpoint to complete and the do another checkpoint and wait for it to finish before returning.
        Returns:
        true if checkpoint is successful, Will return false if wait is false and the routine finds another thread executing a checkpoint.
        Throws:
        StandardException - - encounter exception while doing checkpoint.

      • flush

        public void flush(LogInstant where)
           throws StandardException
        Flush all unwritten log record up to the log instance indicated to disk and sync. Also check to see if database is frozen or corrupt.

        MT - not needed, wrapper method

        Specified by:
        flush in interface LogFactory
        Parameters:
        where - flush log up to here
        Throws:
        StandardException - Standard Derby error policy

      • flushAll

        public void flushAll()
              throws StandardException
        Flush all unwritten log record to disk and sync. Also check to see if database is frozen or corrupt.

        MT - not needed, wrapper method

        Throws:
        StandardException - Standard Derby error policy

      • switchLogFile

        public void switchLogFile()
                   throws StandardException
        Switch to the next log file if possible.

        MT - log factory is single threaded thru a log file switch, the log is frozen for the duration of the switch

        Throws:
        StandardException

      • getCanonicalLogPath

        public java.lang.String getCanonicalLogPath()
        Description copied from interface: LogFactory
        Return the canonical directory of the PARENT of the log directory. The log directory live in the "log" subdirectory of this path. If the log is at the default location (underneath the database directory), this returns null. Should only be called after the log factory is booted.
        Specified by:
        getCanonicalLogPath in interface LogFactory

      • canSupport

        public boolean canSupport(java.util.Properties startParams)
        Description copied from interface: ModuleSupportable
        See if this implementation can support any attributes that are listed in properties. This call may be made on a newly created instance before the boot() method has been called, or after the boot method has been called for a running module.

        The module can check for attributes in the properties to see if it can fulfill the required behaviour. E.g. the raw store may define an attribute called RawStore.Recoverable. If a temporary raw store is required the property RawStore.recoverable=false would be added to the properties before calling bootServiceModule. If a raw store cannot support this attribute its canSupport method would return null. Also see the Monitor class's prologue to see how the identifier is used in looking up properties.
        Actually a better way maybe to have properties of the form RawStore.Attributes.mandatory=recoverable,smallfootprint and RawStore.Attributes.requested=oltp,fast

        Specified by:
        canSupport in interface ModuleSupportable
        Returns:
        true if this instance can be used, false otherwise.

      • stop

        public void stop()
        Stop the log factory

        MT- caller provide synchronization (RESOLVE: this should be called AFTER dataFactory and transFactory are stopped)

        Specified by:
        stop in interface ModuleControl
        See Also:
        Monitor, ModuleFactory

      • serviceASAP

        public boolean serviceASAP()
        Description copied from interface: Serviceable
        If this work should be done as soon as possible, then return true. If it doesn't make any difference if it is done sooner rather than later, then return false. The difference is whether or not the daemon service will be notified to work on this when this work is enqueued or subscribed, in case the serviceable work is put together but not sent to the daemon service directly, like in post commit processing

        MT - MT safe

        Specified by:
        serviceASAP in interface Serviceable

      • serviceImmediately

        public boolean serviceImmediately()
        Description copied from interface: Serviceable
        If this work should be done immediately on the user thread then return true. If it doesn't make any difference if this work is done on a the user thread immediately or if it is performed by another thread asynchronously later, then return false.
        Specified by:
        serviceImmediately in interface Serviceable

      • performWork

        public int performWork(ContextManager context)
        Description copied from interface: Serviceable
        Do whatever it is that you want the daemon to do for you. There may be multiple daemon objects on different thread calling performWork at the same time. The DaemonService will always call performWork with a context manager set up. the DaemonService will clean up the context if an exception is thrown. However, it is up to performWork to manage its own transaction. If you start a transaction in performWork, you must commit or abort it at the end. You may leave the transaction open so that other serviceable may use the transaction and context without starting a new one. On the same token, there may already be an opened transaction on the context. Serviceable performWork should always check the state of the context before use. A Serviceable object should be well behaved while it is performing the daemon work, i.e., it should not take too many resources or hog the CPU for too long or deadlock with anyone else.
        Specified by:
        performWork in interface Serviceable
        Parameters:
        context - the contextManager set up by the DaemonService. There may or may not be the necessary context on it, depending on which other Serviceable object it has done work for.
        Returns:
        the return status is only significant if the Serviceable client was enqueued instead of subscribed. For subscribed client, the return status is ignored. For enqueue client, it returns DONE or REQUEUE. If a REQUEUEd is returned, it would be desirable if this should not be serviceASAP, although no harm is done if this still maintains that this should be serviced ASAP ...

      • appendLogRecord

        public long appendLogRecord(byte[] data,
                            int offset,
                            int length,
                            byte[] optionalData,
                            int optionalDataOffset,
                            int optionalDataLength)
                     throws StandardException
        Append length bytes of data to the log prepended by a long log instant and followed by 4 bytes of length information.

        This method is synchronized to ensure log records are added sequentially to the end of the log.

        MT- single threaded through this log factory. Log records are appended one at a time.

        Throws:
        StandardException - Log Full.

      • openForwardsFlushedScan

        public LogScan openForwardsFlushedScan(LogInstant startAt)
                                throws StandardException
        Open a forward scan of the transaction log.

        MT- read only

        Specified by:
        openForwardsFlushedScan in interface LogFactory
        Parameters:
        startAt - - the LogInstant where we start our scan. null means start at the beginning of the log. This function raises an error if startAt is a LogInstant which is not in the log.
        Returns:
        the LogScan.
        Throws:
        StandardException - Standard Derby exception policy

      • openForwardsScan

        public LogScan openForwardsScan(LogInstant startAt,
                                LogInstant stopAt)
                         throws StandardException
        Get a forwards scan
        Specified by:
        openForwardsScan in interface LogFactory
        Parameters:
        startAt - - the LogInstant where we start our scan. null means start at the beginning of the log. This function raises an error if startAt is a LogInstant which is not in the log.
        stopAt - - the LogInstant where we stop our scan. null means stop at the end of the log. This function raises an error if stopAt is a LogInstant which is not in the log.
        Returns:
        the LogScan.
        Throws:
        StandardException - Standard Derby error policy

      • databaseEncrypted

        public final boolean databaseEncrypted()

      • setDatabaseEncrypted

        public void setDatabaseEncrypted(boolean isEncrypted,
                                 boolean flushLog)
                          throws StandardException
        Sets whether the database is encrypted, all the transaction log has to be encrypted, and flush the log if requested.

        Log needs to be flushed first if the cryptographic state of the database changes (for instance re-encryption with a new key).

        Specified by:
        setDatabaseEncrypted in interface LogFactory
        Parameters:
        isEncrypted - true if the database is encrypted, false if not
        flushLog - true if log needs to be flushed, false otherwise
        Throws:
        StandardException

      • getEncryptionBlockSize

        public int getEncryptionBlockSize()
        return the encryption block size used during encrypted db creation

      • getEncryptedDataLength

        public int getEncryptedDataLength(int length)
        returns the length that will make the data to be multiple of encryption block size based on the given length. Block cipher algorithms like DES and Blowfish ..etc require their input to be an exact multiple of the block size.

      • getFirstUnflushedInstant

        public LogInstant getFirstUnflushedInstant()
        Get the instant of the first record which was not flushed.

        This only works after running recovery the first time.

        MT - RESOLVE:

        Specified by:
        getFirstUnflushedInstant in interface LogFactory

      • getFirstUnflushedInstantAsLong

        public long getFirstUnflushedInstantAsLong()
        Description copied from interface: LogFactory
        Get the log instant long value of the first log record that has not been flushed. Only works after recover() has finished, or (if in slave replication mode) after calling initializeReplicationSlaveRole.
        Specified by:
        getFirstUnflushedInstantAsLong in interface LogFactory
        Returns:
        the log instant long value of the first log record that has not been flushed

      • logArchived

        public boolean logArchived()
        Backup restore - is the log being archived to some directory? if log archive mode is enabled return true else false
        Specified by:
        logArchived in interface LogFactory
        Returns:
        true if the log is being archived.

      • checkVersion

        public boolean checkVersion(int requiredMajorVersion,
                            int requiredMinorVersion,
                            java.lang.String feature)
                     throws StandardException
        Check to see if a database has been upgraded to the required level in order to use a store feature.
        Specified by:
        checkVersion in interface LogFactory
        Parameters:
        requiredMajorVersion - required database Engine major version
        requiredMinorVersion - required database Engine minor version
        feature - Non-null to throw an exception, null to return the state of the version match.
        Returns:
        true if the database has been upgraded to the required level, false otherwise.
        Throws:
        StandardException - if the database is not at the require version when feature feature is not null .

      • inReplicationMasterMode

        public boolean inReplicationMasterMode()
        Used to determine if the replication master mode has been started, and the logging for unlogged operations needs to be enabled.
        Specified by:
        inReplicationMasterMode in interface LogFactory
        Returns:
        true If the master replication mode is turned on and the unlogged operations need to be logged. false If the master replication mode is turned off and the unlogged operations need not be logged.

      • inRFR

        public boolean inRFR()
        Specified by:
        inRFR in interface LogFactory

      • checkpointInRFR

        public void checkpointInRFR(LogInstant cinstant,
                            long redoLWM,
                            long undoLWM,
                            DataFactory df)
                     throws StandardException
        redo a checkpoint during rollforward recovery
        Specified by:
        checkpointInRFR in interface LogFactory
        Parameters:
        cinstant - The LogInstant of the checkpoint
        redoLWM - Redo Low Water Mark in the check point record
        undoLWM - Undo Low Water Mark in the checkpoint
        df - - the data factory
        Throws:
        StandardException

      • startReplicationMasterRole

        public void startReplicationMasterRole(MasterFactory masterFactory)
                                throws StandardException
        Make this LogFactory pass log records to the MasterFactory every time a log record is appended to the log on disk, and notify the MasterFactory when a log disk flush has taken place.
        Specified by:
        startReplicationMasterRole in interface LogFactory
        Parameters:
        masterFactory - The MasterFactory service responsible for controlling the master side replication behaviour.
        Throws:
        StandardException - Standard Derby exception policy, thrown on replication startup error. Will only be thrown if replication is attempted started on a readonly database, i.e, never thrown here.

      • stopReplicationMasterRole

        public void stopReplicationMasterRole()
        Stop this LogFactory from passing log records to the MasterFactory and from notifying the MasterFactory when a log disk flush has taken place.
        Specified by:
        stopReplicationMasterRole in interface LogFactory

      • stopReplicationSlaveRole

        public void stopReplicationSlaveRole()
                              throws StandardException
        Stop the slave functionality for this LogFactory. Calling this method causes the thread currently doing recovery to stop the recovery process and throw a StandardException with SQLState SHUTDOWN_DATABASE. This should only be done when the database will be shutdown.
        Throws:
        StandardException - Standard Derby exception policy
        See Also:
        SlaveDatabase

      • initializeReplicationSlaveRole

        public void initializeReplicationSlaveRole()
                                    throws StandardException
        Initializes logOut so that log received from the replication master can be appended to the log file. Normally, logOut (the file log records are appended to) is set up as part of the recovery process. When the database is booted in replication slave mode, however, recovery will not get to the point where logOut is initialized until this database is no longer in slave mode. Since logOut is needed to append log records received from the master, logOut needs to be set up for replication slave mode. This method finds the last log record in the log file with the highest number. logOut is set up so that log records will be appended to the end of that file, and the endPosition and lastFlush variables are set to point to the end of the same file. All this is normally done as part of recovery. After the first log file switch resulting from applying log received from the master, recovery will be allowed to read up to, but not including, the current log file which is the file numbered logFileNumber. Note that this method must not be called until LogToFile#boot() has completed. Currently, this is ensured because RawStore#boot starts the SlaveFactory (in turn calling this method) after LogFactory.boot() has completed. Race conditions for logFileNumber may occur if this is changed.
        Throws:
        StandardException - Standard Derby error policy

      • failoverSlave

        public void failoverSlave()
        Used to make the slave stop appending log records, complete recovery and boot the database.

      • run

        public final java.lang.Object run()
                           throws java.io.IOException,
                                  StandardException
        Specified by:
        run in interface java.security.PrivilegedExceptionAction<java.lang.Object>
        Throws:
        java.io.IOException
        StandardException
Skip navigation links

DataMelt 3.0 © DataMelt by jWork.ORG

You see the box below because you did not login.