[Phanx]

Numbered channels (including custom ones) are part of the "CHANNEL" chat type. You'd need to look at the fourth argument to SendChatMessage (the actual channel, by number, for "CHANNEL" messages, or the recipient's name for "WHISPER" messages) in addition to the second one (the chat type). Your current code mistakenly names the second argument "channel"; you should always check the API documentation and try to stick with the "official" argument names, to avoid confusion.

local chatTypesToProcess = {
    ["GUILD"] = true,
    ["SAY"] = true,
    ["YELL"] = true,
    ["CHANNEL"] = true,
}
 
local channelsToProcess = {
    ["RP"] = true,
}
 
local old = SendChatMessage
 
SendChatMessage = function(message, chatType, language, channel)
    if chatType and chatTypesToProcess[chatType] and message:len() > 0 and not message:match("^[%(*/]") then
        if chatType == "CHANNEL" then
            -- Get the channel's name from its number. We don't want to
            -- use channel numbers in the list, because they can change.
            local _, channelName = GetChannelName(channel)
            if channelName and channelsToProcess[channelName:upper()] then
                -- We need to uppercase the channel's name, because the
                -- capitalization can change too.
                message = message:gsub("r", "rr"):gsub("rrrr", "rr"):gsub("R", "RR"):gsub("RRRR", "RR")
            end
        else
            message = message:gsub("r", "rr"):gsub("rrrr", "rr"):gsub("R", "RR"):gsub("RRRR", "RR")
        end
    end
    old(message, chatType, language, channel)
end

See also:
http://www.wowpedia.org/API_SendChatMessage
http://www.wowpedia.org/API_GetChannelName