Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PubSub api #714

Merged
merged 58 commits into from
Jul 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
a280504
Add redis pubsub layer
0pg Dec 28, 2022
cd4a3f3
Fix PubSub api
0pg Feb 7, 2023
a01213d
Apply changes to t/c
0pg Feb 7, 2023
9541519
Set private accessor
0pg Feb 7, 2023
ba89a59
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Feb 7, 2023
9723182
Fix cleanup code
0pg Feb 7, 2023
65b524b
Modify callback and unsubscribe logic in SingleNodeRedisPubSub
0pg Feb 7, 2023
892ba20
Apply changes to t/c
0pg Feb 7, 2023
43edb47
Modify PubSub api
0pg Feb 7, 2023
a6f74fa
Apply changes to t/c
0pg Feb 7, 2023
c7decb7
Fix typo
0pg Feb 7, 2023
b6438f0
Modify accessor
0pg Feb 7, 2023
5fb75e4
Update redis/src/main/scala/zio/redis/Output.scala
0pg Feb 9, 2023
ce97f95
Update redis/src/main/scala/zio/redis/Output.scala
0pg Feb 9, 2023
a25805b
Modify responsibility to transform streams
0pg Feb 9, 2023
fd0b303
Apply changes to t/c
0pg Feb 9, 2023
7cb1491
Move SubscriptionKey into only used place
0pg Feb 10, 2023
2ab2db1
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Feb 10, 2023
87eee71
Fix onSubscribe callback race condition bug
0pg Feb 11, 2023
f532e4b
Add pacakge private accessor to PubSubCommand
0pg Feb 11, 2023
e9e2a3c
Modify unsubscrption release logic
0pg Feb 11, 2023
014d2e4
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Feb 14, 2023
49c141b
Modify RedisPubSubCommand interface
0pg Feb 14, 2023
8b6ce73
Apply changes to test
0pg Feb 14, 2023
f44c7fc
Add pubsub layer in Main
0pg Feb 14, 2023
758ea9b
Remove BinaryCodec layer to RedisPubSub interface
0pg Feb 20, 2023
9febcf8
Modify SingleNodeRedisPubSub failover step
0pg Feb 20, 2023
92aea5e
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Mar 26, 2023
3fc77c0
Separate Publish and Subscribe
0pg Mar 30, 2023
3c8a307
Rename classes
0pg Mar 30, 2023
0ea75a1
Reduce duplication
0pg Mar 31, 2023
ef3b9f1
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Mar 31, 2023
478577b
Apply upstream changes
0pg Mar 31, 2023
c4105ff
Replace Hub to Queue
0pg Apr 4, 2023
f432815
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg Apr 25, 2023
149ae4b
Remove unused code
0pg Apr 27, 2023
67652e6
Refine subscription api
0pg Apr 27, 2023
8606409
Rename fields
0pg May 2, 2023
6a3f68f
Update modules/redis/src/main/scala/zio/redis/RedisSubscription.scala
0pg May 2, 2023
1a0862f
Add newline
0pg May 5, 2023
92457c4
Fix broken compile
0pg May 7, 2023
bf52e26
Formatting
0pg May 7, 2023
c027295
Fix lint
0pg May 7, 2023
a154683
Reduce duplications and refactor package layout
0pg May 10, 2023
592d327
Commit suggestion
0pg May 10, 2023
386fefd
Fix broken compile
0pg May 23, 2023
415f75e
Merge remote-tracking branch 'upstream/master' into impl/pubsub
0pg May 23, 2023
1699211
Use hub instead of chunks of queue
0pg May 30, 2023
be11d67
Add release on error
0pg May 30, 2023
552e002
Extraact common logic
0pg May 30, 2023
e218b08
Ensure order of subs/unsubs
0pg May 31, 2023
ac85cb9
Update modules/redis/src/main/scala/zio/redis/internal/SingleNodeSubs…
0pg Jun 7, 2023
dadb5e7
Use ConcurrentMap instead of Map of Ref
0pg Jun 7, 2023
88c3f4e
Add private accessor
0pg Jun 7, 2023
6e50d4d
Fix accessor
0pg Jun 7, 2023
5ca98e7
Add doc
0pg Jun 26, 2023
539ef3f
Add missing params
0pg Jun 26, 2023
63cc263
Fix test codes
0pg Jul 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ lazy val redis =
.settings(
libraryDependencies ++= List(
"dev.zio" %% "zio-streams" % zioVersion,
"dev.zio" %% "zio-concurrent" % zioVersion,
"dev.zio" %% "zio-schema" % zioSchemaVersion,
"dev.zio" %% "zio-schema-protobuf" % zioSchemaVersion % Test,
"dev.zio" %% "zio-test" % zioVersion % Test,
Expand Down
49 changes: 49 additions & 0 deletions modules/redis/src/main/scala/zio/redis/Output.scala
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package zio.redis

import zio._
import zio.redis.internal.PubSub.{PushMessage, SubscriptionKey}
import zio.redis.internal.RespValue
import zio.redis.options.Cluster.{Node, Partition, SlotRange}
import zio.schema.Schema
Expand Down Expand Up @@ -639,6 +640,54 @@ object Output {
}
}

private[redis] case object PushMessageOutput extends Output[PushMessage] {
protected def tryDecode(respValue: RespValue): PushMessage =
respValue match {
case RespValue.NullArray => throw ProtocolError(s"Array must not be empty")
case RespValue.Array(values) =>
val name = MultiStringOutput.unsafeDecode(values(0))
val key = MultiStringOutput.unsafeDecode(values(1))
name match {
case "subscribe" =>
val num = LongOutput.unsafeDecode(values(2))
PushMessage.Subscribed(SubscriptionKey.Channel(key), num)
case "psubscribe" =>
val num = LongOutput.unsafeDecode(values(2))
PushMessage.Subscribed(SubscriptionKey.Pattern(key), num)
case "unsubscribe" =>
val num = LongOutput.unsafeDecode(values(2))
PushMessage.Unsubscribed(SubscriptionKey.Channel(key), num)
case "punsubscribe" =>
val num = LongOutput.unsafeDecode(values(2))
PushMessage.Unsubscribed(SubscriptionKey.Pattern(key), num)
case "message" =>
val message = values(2)
PushMessage.Message(SubscriptionKey.Channel(key), key, message)
case "pmessage" =>
val channel = MultiStringOutput.unsafeDecode(values(2))
val message = values(3)
PushMessage.Message(SubscriptionKey.Pattern(key), channel, message)
case other => throw ProtocolError(s"$other isn't a pushed message")
}
case other => throw ProtocolError(s"$other isn't an array")
}
}

case object NumSubResponseOutput extends Output[Map[String, Long]] {
protected def tryDecode(respValue: RespValue): Map[String, Long] =
respValue match {
case RespValue.Array(values) =>
val builder = Map.newBuilder[String, Long]
values.grouped(2).foreach { chunk =>
val channel = MultiStringOutput.unsafeDecode(chunk(0))
val numOfSubs = LongOutput.unsafeDecode(chunk(1))
builder += channel -> numOfSubs
}
builder.result()
case other => throw ProtocolError(s"$other isn't an array")
}
}

private def decodeDouble(bytes: Chunk[Byte]): Double = {
val text = new String(bytes.toArray, StandardCharsets.UTF_8)
try text.toDouble
Expand Down
1 change: 1 addition & 0 deletions modules/redis/src/main/scala/zio/redis/Redis.scala
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ trait Redis
with api.Streams
with api.Scripting
with api.Cluster
with api.Publishing

object Redis {
lazy val cluster: ZLayer[CodecSupplier & RedisClusterConfig, RedisError, Redis] =
Expand Down
12 changes: 11 additions & 1 deletion modules/redis/src/main/scala/zio/redis/RedisError.scala
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

package zio.redis

import zio.redis.internal.PubSub.SubscriptionKey
import zio.redis.internal.RespCommand
import zio.redis.options.Cluster.Slot

import java.io.IOException
Expand Down Expand Up @@ -46,5 +48,13 @@ object RedisError {
object Moved {
def apply(slotAndAddress: (Slot, RedisUri)): Moved = Moved(slotAndAddress._1, slotAndAddress._2)
}
final case class IOError(exception: IOException) extends RedisError
final case class IOError(exception: IOException) extends RedisError
final case class CommandNameNotFound(message: String) extends RedisError
object CommandNameNotFound {
def apply(command: RespCommand): CommandNameNotFound = CommandNameNotFound(command.args.toString())
}

sealed trait PubSubError extends RedisError
final case class InvalidPubSubCommand(command: String) extends PubSubError
final case class SubscriptionStreamAlreadyClosed(key: SubscriptionKey) extends PubSubError
}
35 changes: 35 additions & 0 deletions modules/redis/src/main/scala/zio/redis/RedisSubscription.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Copyright 2021 John A. De Goes and the ZIO contributors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package zio.redis

import zio._
import zio.redis.internal.SubscriptionExecutor

trait RedisSubscription extends api.Subscription
mijicd marked this conversation as resolved.
Show resolved Hide resolved

object RedisSubscription {
lazy val local: ZLayer[CodecSupplier, RedisError.IOError, RedisSubscription] =
SubscriptionExecutor.local >>> makeLayer

lazy val singleNode: ZLayer[CodecSupplier & RedisConfig, RedisError.IOError, RedisSubscription] =
SubscriptionExecutor.layer >>> makeLayer

private def makeLayer: URLayer[CodecSupplier & SubscriptionExecutor, RedisSubscription] =
ZLayer.fromFunction(Live.apply _)

private final case class Live(codecSupplier: CodecSupplier, executor: SubscriptionExecutor) extends RedisSubscription
}
5 changes: 5 additions & 0 deletions modules/redis/src/main/scala/zio/redis/ResultBuilder.scala
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package zio.redis
import zio.IO
import zio.redis.ResultBuilder.NeedsReturnType
import zio.schema.Schema
import zio.stream.Stream

sealed trait ResultBuilder {
final def map(f: Nothing => Any)(implicit nrt: NeedsReturnType): IO[Nothing, Nothing] = ???
Expand All @@ -45,4 +46,8 @@ object ResultBuilder {
trait ResultOutputBuilder extends ResultBuilder {
def returning[R: Output]: IO[RedisError, R]
}

trait ResultStreamBuilder1[+F[_]] extends ResultBuilder {
def returning[R: Schema]: Stream[RedisError, F[R]]
}
}
89 changes: 89 additions & 0 deletions modules/redis/src/main/scala/zio/redis/api/Publishing.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
/*
* Copyright 2021 John A. De Goes and the ZIO contributors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package zio.redis.api

import zio.redis.Input._
import zio.redis.Output._
import zio.redis._
import zio.redis.internal.{RedisCommand, RedisEnvironment}
import zio.schema.Schema
import zio.{Chunk, IO}

trait Publishing extends RedisEnvironment {
import Publishing._

/**
* Posts a message to the given channel.
*
* @param channel
* Target channel name for publishing messages.
* @param message
* The value of the message to be published to the channel.
* @return
* Returns the number of clients that received the message.
*/
final def publish[A: Schema](channel: String, message: A): IO[RedisError, Long] = {
val command = RedisCommand(Publish, Tuple2(StringInput, ArbitraryKeyInput[A]()), LongOutput, executor)
command.run((channel, message))
}

/**
* Lists the currently active channel that has one or more subscribers (excluding clients subscribed to patterns).
*
* @param pattern
* Pattern to get matching channels.
* @return
* Returns a list of active channels matching the specified pattern.
*/
final def pubSubChannels(pattern: String): IO[RedisError, Chunk[String]] = {
val command = RedisCommand(PubSubChannels, StringInput, ChunkOutput(MultiStringOutput), executor)
command.run(pattern)
}

/**
* The number of unique patterns that are subscribed to by clients.
*
* @return
* Returns the number of patterns all the clients are subscribed to.
*/
final def pubSubNumPat: IO[RedisError, Long] = {
val command = RedisCommand(PubSubNumPat, NoInput, LongOutput, executor)
command.run(())
}

/**
* The number of subscribers (exclusive of clients subscribed to patterns) for the specified channels.
*
* @param channel
* Channel name to get the number of subscribers.
* @param channels
* Channel names to get the number of subscribers.
* @return
* Returns a map of channel and number of subscribers for channel.
*/
final def pubSubNumSub(channel: String, channels: String*): IO[RedisError, Map[String, Long]] = {
val command = RedisCommand(PubSubNumSub, NonEmptyList(StringInput), NumSubResponseOutput, executor)
command.run((channel, channels.toList))
}
}

private[redis] object Publishing {
final val Publish = "PUBLISH"
final val PubSubChannels = "PUBSUB CHANNELS"
final val PubSubNumPat = "PUBSUB NUMPAT"
final val PubSubNumSub = "PUBSUB NUMSUB"
}
Loading
Loading